diff --git a/src/about/whats-new-23.md b/src/about/whats-new-23.md index ac1d6ab2..cb27d4f3 100644 --- a/src/about/whats-new-23.md +++ b/src/about/whats-new-23.md @@ -8,7 +8,7 @@ DataJoint 2.3 introduces the **provenance trinity** — `Diagram.trace`, `self.u ## Overview -DataJoint's provenance guarantee rests on the convention that `make(self, key)` reads only from declared upstream dependencies and writes only to `self` (and its Parts). The framework has always *defined* this boundary but never *checked* it: a `make()` could `fetch()` from any table, making the dependency invisible to the foreign-key graph and silently breaking the provenance claim downstream. +DataJoint's provenance model rests on the convention that `make(self, key)` reads only from declared upstream dependencies and writes only to `self` (and its Parts). The framework has always *defined* this boundary but never *checked* it: a `make()` could `fetch()` from any table, making the dependency invisible to the foreign-key graph and silently breaking the provenance claim downstream. The 2.3 trinity closes that loop with three pieces designed as a unit: diff --git a/src/how-to/run-computations.md b/src/how-to/run-computations.md index b925d116..ff03e9e0 100644 --- a/src/how-to/run-computations.md +++ b/src/how-to/run-computations.md @@ -246,8 +246,48 @@ class FilteredComputation(dj.Computed): | `reserve_jobs` | `False` | Reserve jobs for distributed computing | | `suppress_errors` | `False` | Continue on errors | +## Provenance-safe make() (strict provenance) + +A `make()` is provenance-safe when it reads only from its declared upstream dependencies and writes only to `self` (and its Parts). DataJoint 2.3 gives you an ergonomic read channel and an opt-in runtime guardrail to keep it that way. + +### Read ancestors through `self.upstream` + +Inside `make()`, `self.upstream[T]` returns the ancestor table `T` pre-restricted to the rows that contributed to the current `key`: + +```python +def make(self, key): + # Reads pre-restricted to the current key — no manual `& key` + rate = self.upstream[Recording].fetch1('sampling_rate') + samples = self.upstream[Recording].to_arrays('signal') + + result = compute_spectrum(samples, rate) + self.insert1({**key, 'spectrum': result}) +``` + +`self.upstream` exposes declared ancestors only. To read your table's **own** Parts mid-`make()`, use `self.PartName` directly — that is permitted under strict mode. + +### Roll out `strict_provenance` in stages + +`dj.config["strict_provenance"]` defaults to `False`, so nothing changes until you opt in. Adopt it incrementally: + +1. **Enable it in staging** and run your pipeline: + + ```python + dj.config["strict_provenance"] = True + ``` + +2. **Fix the undeclared reads the guardrail surfaces.** Each violation raises a `DataJointError` naming the offending table. Either route the read through `self.upstream[T]` (if `T` is a declared ancestor) or declare the missing foreign-key dependency. +3. **Enable it in production** once staging is clean. + +### What the guardrail does and does not catch + +When `strict_provenance=True`, DataJoint checks — through the Python client — that a `make()` reads only from declared ancestors and writes (`insert`/`insert1`/`update1`) only to `self` and its Parts, with each written row's key consistent with the current `key`. + +It is a **best-effort development guardrail, not an airtight boundary.** Raw SQL, other clients, existence/count idioms (`len(q)`, `bool(q)`, `key in q`), restriction-by-table, and `delete()` are **not** intercepted. See the [Provenance Spec — Enforcement model and its limits](../reference/specs/provenance.md#enforcement-model-and-its-limits) for the full list, and the [Provenance Specification](../reference/specs/provenance.md) for the migration path in detail. + ## See Also - [Computation Model](../explanation/computation-model.md/) — How computation works +- [Provenance Specification](../reference/specs/provenance.md) — `Diagram.trace`, `self.upstream`, `strict_provenance` - [Distributed Computing](distributed-computing.md) — Multi-worker setup - [Handle Errors](handle-errors.md) — Error recovery diff --git a/src/reference/specs/autopopulate.md b/src/reference/specs/autopopulate.md index dc13d1d6..2027ab5a 100644 --- a/src/reference/specs/autopopulate.md +++ b/src/reference/specs/autopopulate.md @@ -317,7 +317,28 @@ def make(self, key): **Upstream-only convention:** Inside `make()`, fetch only from tables that are strictly upstream in the pipeline—tables referenced by foreign keys in the definition, their ancestors, and their part tables. This ensures reproducibility: computed results depend only on their declared dependencies. -This convention is not currently enforced programmatically but is critical for pipeline integrity. Some pipelines violate this rule for operational reasons, which makes them non-reproducible. A future release may programmatically enforce upstream-only fetches inside `make()`. +As of DataJoint 2.3, this convention is checked (opt-in) at runtime: setting `dj.config["strict_provenance"] = True` makes reads from undeclared tables and writes to tables other than `self` (and its Parts) raise `DataJointError` inside `make()`. The check is a best-effort development guardrail; see the [Provenance Specification](provenance.md) for the enforcement model and its documented limits. + +### 4.3.1 `self.upstream`: Pre-Restricted Ancestor Access + +Inside `make()`, the framework provides `self.upstream` — a pre-constructed `Diagram.trace(self & key)` — as the ergonomic way to read declared ancestors without manual `& key` restriction: + +```python +def make(self, key): + # Each ancestor arrives pre-restricted to the rows that contributed to this key + rate = self.upstream[Recording].fetch1('sampling_rate') + trials = self.upstream[Trial].to_dicts() + self.insert1({**key, 'result': compute(trials, rate)}) +``` + +Behavior: + +- **Lifecycle:** `self.upstream` is set to `Diagram.trace(self & key)` immediately before `make()` is invoked and cleared afterward — including when `make()` raises. Accessing it outside `make()` raises `DataJointError`. +- **Lazy:** the trace diagram is built once per `make()` call; no SQL fires until an ancestor is accessed and fetched. Fetch results are not cached — reading the same ancestor twice issues two queries. +- **Tripartite:** `self.upstream` is available across all tripartite phases (`make_fetch`, `make_compute`, `make_insert`) of the same `make()` call. +- **Scope:** it exposes declared ancestors only. A table's own Parts are descendants, not ancestors — read them directly as `self.PartName`. + +See [Provenance Specification §2](provenance.md#2-selfupstream-inside-make) for the normative semantics. ### 4.4 Tripartite Make Pattern diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index 36414c2c..13c8d9c6 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -33,7 +33,7 @@ When propagating a restriction across an edge `parent → child`, one of three r | Rule | Trigger | Effect on child | |---|---|---| -| **F1. Copy** | `not aliased and parent_attrs ⊆ child_pk` | Child inherits the parent's restriction directly (same attribute names; literal restriction values copy as-is). | +| **F1. Copy** | `not aliased and parent_attrs` **non-empty** `and parent_attrs ⊆ child_pk` | Child inherits the parent's restriction directly (same attribute names; literal restriction values copy as-is). An empty attribute set takes rule 3. | | **F2. Aliased rename** | `aliased` | Child's restriction is `parent.proj(**{fk_col: parent_col for fk_col, parent_col in attr_map.items()})` — the parent expression with columns renamed to match the child's column names. | | **F3. Project** | `not aliased and parent_attrs ⊄ child_pk` | Child's restriction is `parent.proj()` — the parent projected to its primary key. | @@ -45,9 +45,9 @@ Symmetric inverses of the forward rules. Used by `part_integrity="cascade"` to p | Rule | Trigger | Effect on parent | |---|---|---| -| **U1. Copy** | `not aliased and child_attrs ⊆ parent_pk` | Parent inherits the child's restriction directly (shared attribute names). | +| **U1. Copy** | `not aliased and child_attrs` **non-empty** `and child_attrs ⊆ parent_pk` | Parent inherits the child's restriction directly (shared attribute names). An empty attribute set takes rule 3. | | **U2. Aliased reverse-rename** | `aliased` | Parent's restriction is `child.proj(**{parent_col: fk_col for fk_col, parent_col in attr_map.items()})` — the child expression with FK columns renamed back to the parent's column names. | -| **U3. Project** | `not aliased and child_attrs ⊄ parent_pk` | Parent's restriction is `child.proj()` — the child projected to attributes in the parent's primary key. | +| **U3. Project** | `not aliased and child_attrs ⊄ parent_pk` | Parent's restriction is `child.proj(*attr_map.keys())` — the child projected onto its FK columns (which, when non-aliased, share names with the parent's PK) so the parent restriction joins on the right columns. | ## Cascade flow @@ -71,7 +71,7 @@ Master-Part integrity reflects the contract that a Master row exists for every P | Mode | Behavior | |---|---| -| `"enforce"` (default) | If a delete would remove a Part row without removing the corresponding Master row, the entire delete is rolled back with `DataJointError`. The Master row is checked **after** the delete; the integrity violation is detected post-hoc and reversed. | +| `"enforce"` (default) | If a delete would remove a Part row without removing the corresponding Master row, the entire delete is rolled back with `DataJointError`. The Master row is checked **after** the delete; the integrity violation is detected post-hoc and reversed. The *intent* is row-level (each deleted Part row should have its Master deleted), but the shipped post-check is **table-level** — see [Limitations](#limitations) for the resulting rare false negatives and false positives. | | `"ignore"` | No upward propagation; no post-check. Use when the Master row is intentionally preserved and the user has accepted that Part rows may be orphaned. Caller is responsible for the consequences. | | `"cascade"` | **Upward propagation enabled.** When cascade reaches a Part, the Master is also restricted (via the upward rules below), and the Master then forward-cascades back down to **all** its Parts (siblings of the originating Part included). Used when the user wants the master-part group treated atomically. | @@ -145,7 +145,7 @@ When `(Subject.Recording & {"recording_id": 5}).delete(part_integrity="cascade") 2. **FK path.** `shortest_path(Subject, Recording) = [Subject, Subject.Session, alias_node, Subject.Recording]` → real path `[Subject, Subject.Session, Subject.Recording]`. 3. **Walk reversed.** - Edge `Subject.Session → Subject.Recording`: `aliased=True`. Apply **U2** — `Subject.Session` is restricted by `Subject.Recording.proj(subject_id='src_subject', session_id='src_session')`. - - Edge `Subject → Subject.Session`: `aliased=False`, `child_attrs={subject_id, session_id} ⊆ parent_pk={subject_id}`? No (`session_id` not in parent pk). Apply **U3** — `Subject` is restricted by `Subject.Session.proj()` (projected to `subject_id`). + - Edge `Subject → Subject.Session`: `aliased=False`, `child_attrs={subject_id, session_id} ⊆ parent_pk={subject_id}`? No (`session_id` not in parent pk). Apply **U3** — `Subject` is restricted by `Subject.Session.proj(*attr_map.keys())`, projecting the child onto its FK columns; for this primary FK those columns are just `subject_id`, so this is equivalent to `Subject.Session.proj()` projected to `subject_id`. 4. **Materialize Master.** `Subject`'s restriction is fetched into a value tuple; replaces the chained `QueryExpression`. 5. **Forward pass.** Master forward-cascades back down to `Subject.Session` and `Subject.Recording` (and any sibling Parts not on the original path), now with the materialized restriction. @@ -191,10 +191,19 @@ For a cascade subgraph with N nodes and E edges, propagation runs in at most O(N ## What is not part of this specification -- **`Diagram.trace()`** for general upstream restriction propagation: a related but distinct feature tracked in [#1423](https://github.com/datajoint/datajoint-python/issues/1423). `trace()` exposes upstream propagation as a first-class operator; the cascade engine's upward walk above is a closed implementation detail of `part_integrity="cascade"`. +- **`Diagram.trace()`** for general upstream restriction propagation: a related but distinct feature that **shipped in 2.3** and reuses the same upward rules (U1/U2/U3) defined above. `trace()` exposes upstream propagation as a first-class operator; the cascade engine's upward walk in this document is the same machinery applied inside `part_integrity="cascade"`. See the [Provenance Specification](provenance.md) for `trace`'s API and semantics. - **Custom propagation rules** (user-defined): not supported. The three forward and three upward rules cover the cases the FK graph can produce. - **Cross-schema cascade**: handled by `dependencies.load_all_downstream()` called from `Diagram.cascade()`; orthogonal to the propagation rules described here. +## Limitations + +The following are known, documented behaviors of the cascade engine as shipped: + +- **Single FK path (part→master walk).** The upward walk uses `nx.shortest_path` to find the FK chain from a Part to its Master. If a Part reaches its Master through multiple distinct FK chains, restrictions carried by the non-shortest paths are not applied. Workaround: use `part_integrity="ignore"` and perform the additional deletes manually. +- **Materialization memory cost.** The master restriction is materialized via `to_arrays()` (to avoid MySQL error 1093, see [Materialization at the Master](#materialization-at-the-master)). The cost is bounded by the number of distinct master rows referenced by the matching parts. Cascade **preview** (`Diagram.cascade(...).counts()`) pays the same materialization cost as an actual delete. +- **Empty-match sentinel.** When no master rows match, the master carries an always-false restriction and appears with zero rows in `counts()` and iteration. This is by design, not an error. +- **Enforce granularity.** The `part_integrity="enforce"` post-check is **table-level**: it verifies that *some* rows of the master table were deleted whenever part rows were, not that each deleted part row's *specific* master row was deleted. As a result, rare false negatives (an unrelated master row happened to be deleted in the same cascade, masking a genuine orphan) and false positives (deleting already-orphaned part rows whose master was removed earlier) are possible. + ## References - Source: `src/datajoint/diagram.py` — `Diagram.cascade`, `_propagate_restrictions`, `_apply_propagation_rule`, `_apply_propagation_rule_upward`, `_propagate_part_to_master`. diff --git a/src/reference/specs/diagram.md b/src/reference/specs/diagram.md index 30772354..f017bb3f 100644 --- a/src/reference/specs/diagram.md +++ b/src/reference/specs/diagram.md @@ -143,7 +143,7 @@ Create a cascade diagram for delete. Builds a complete dependency graph from the | Value | Behavior | |-------|----------| -| `"enforce"` | Error if parts would be deleted before masters | +| `"enforce"` | Default. The preview itself never errors; master-part integrity is enforced by `Table.delete()`'s post-check, which rolls back the delete if part rows would be removed without their masters (see [Cascade Spec](cascade.md#part_integrity-modes)). | | `"ignore"` | Allow deleting parts without masters | | `"cascade"` | Propagate restriction upward from part to master, then re-propagate downstream to all sibling parts | @@ -154,6 +154,32 @@ With `"cascade"`, the restriction flows **upward** from a part table to its mast dj.Diagram.cascade(Session & {'subject_id': 'M001'}).counts() ``` +`part_integrity` accepts only `"enforce"`, `"ignore"`, or `"cascade"`; any other value raises `ValueError`. + +### `Diagram.trace()` (class method) + +```python +dj.Diagram.trace(table_expr) +``` + +The **upstream mirror of `cascade()`**. Where `cascade()` walks downstream to every descendant, `trace()` walks **upstream** from a (possibly restricted) table expression to every ancestor across all loaded schemas, propagating the restriction along the way. Like `cascade()`, convergence is **OR** — an ancestor is included if reachable through *any* FK path from the seed. + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `table_expr` | QueryExpression | — | A restricted table expression whose ancestry is traced | + +**Returns:** New `Diagram` containing the seed and its ancestors, each pre-restricted through the FK join path. + +Access a pre-restricted ancestor by indexing the trace: + +```python +trace = dj.Diagram.trace(MyChild & key) +session = trace[Session] # Session restricted to ancestors of MyChild & key +session.fetch1('session_date') +``` + +`trace()` reuses the same **upward propagation rules** (U1/U2/U3) documented in the [Cascade Spec](cascade.md#upward-propagation-child-parent). The [Provenance Specification §1](provenance.md#1-diagramtracetable_expr) is the normative spec for its API and semantics; inside `make()`, `self.upstream` is the per-`key` instance of a trace. + ### `restrict()` ```python diff --git a/src/reference/specs/provenance.md b/src/reference/specs/provenance.md index 1c1cc74c..ef047825 100644 --- a/src/reference/specs/provenance.md +++ b/src/reference/specs/provenance.md @@ -15,7 +15,7 @@ For the related downstream operation, see [Cascade Specification](cascade.md). F ## Why this exists -DataJoint's provenance guarantee — that every computed row can be traced back to the exact upstream rows it was derived from — rests on the convention that `make(self, key)` only reads from declared upstream dependencies. Today the framework **defines** this convention but does not **check** it: a `make()` can `fetch()` from any table, making the undeclared dependency invisible to the FK graph and breaking the provenance claim downstream. +DataJoint's provenance model — that every computed row can be traced back to the exact upstream rows it was derived from — rests on the convention that `make(self, key)` only reads from declared upstream dependencies. Today the framework **defines** this convention but does not **check** it: a `make()` can `fetch()` from any table, making the undeclared dependency invisible to the FK graph and breaking the provenance claim downstream. Closing the loop requires three pieces working together: @@ -203,6 +203,11 @@ The construction is **lazy** in the sense that the underlying SQL is not issued Requesting any table outside this set raises `DataJointError` — including tables that exist in the schema but are not ancestors of `self`. This is the same guarantee `Diagram.trace(...)` provides; `self.upstream` is just the per-`make()` instance of it. +`self.upstream` exposes ancestors only; a table's **own** Parts are not reachable via `self.upstream[...]` (they are descendants). Inside `make()`, read your own Parts directly as `self.PartName` — permitted under strict mode. + +!!! note "Merge/master-part boundaries" + Trace (and therefore `self.upstream`) walks ancestor FK edges only. It does **not** descend from an ancestor Master into that Master's Parts — an ancestor's Part is included only when the Part itself lies on the FK path to the seed. In a merge-table shape `Parent → Master.Part → Master → Child`, `trace(Child & key)` reaches `Master` but neither `Master.Part` nor `Parent`. + ### Examples Without `strict_provenance` (the default), `self.upstream` is a convenience — the same data is reachable via `(Upstream & key).fetch1(...)`. The win is ergonomic: @@ -246,7 +251,7 @@ The `key` argument to `make()` — and any `make_kwargs` optionally forwarded th |---|---|---|---| | `strict_provenance` | `bool` | `False` | When `True`, checks upstream-only reads and target-only writes inside `make()`. | -The flag is read from the global `dj.config` at the start of each `make()` invocation and captured into a per-call enforcement context stored in a `contextvars.ContextVar`, so the active context follows the executing task rather than being shared as global mutable state. Because the config value itself is process-global, concurrent populates in the same process cannot run with different `strict_provenance` values — each `make()` sees whatever the config holds when it begins. +The flag is read from the connection's config (`connection._config`, normally the global `dj.config`) at the start of each `make()` invocation and captured into a per-call enforcement context stored in a `contextvars.ContextVar`, so the active context follows the executing task rather than being shared as global mutable state. Because the config value itself is process-global, concurrent populates in the same process cannot run with different `strict_provenance` values — each `make()` sees whatever the config holds when it begins. ### Enforcement model and its limits @@ -256,8 +261,9 @@ They do **not** constitute a security or correctness boundary: - Raw SQL (`connection.query(...)`), a second client or process, a database console, a BI tool, or a non-Python binding all bypass the checks entirely. - Within the Python client, the read check currently allows a direct read of a *declared* ancestor just as it allows reads through `self.upstream` — it flags reads of tables *outside* the allowed set, not the channel used to reach one inside it (see the note under [Read enforcement](#read-enforcement)). -- **Existence and count idioms bypass the read gate.** `len(q)`, `bool(q)`, and `item in q` issue their `COUNT`/`EXISTS` SQL directly rather than through the gated fetch path, so a read of an undeclared table via `len(Undeclared & key)`, `bool(Undeclared & cond)`, or `key in Undeclared` is **not** caught. Only the `fetch`/`fetch1`/`to_arrays`/iteration path is gated. +- **Existence and count idioms bypass the read gate.** `len(q)`, `bool(q)`, and `item in q` issue their `COUNT`/`EXISTS` SQL directly rather than through the gated fetch path, so a read of an undeclared table via `len(Undeclared & key)`, `bool(Undeclared & cond)`, or `key in Undeclared` is **not** caught. Only the `fetch`/`fetch1`/`to_arrays`/iteration path is gated. The same bypass applies to `Aggregation` and `Union` results (e.g. `len(dj.U(...).aggr(Undeclared, ...))` bypasses the gate even though `.fetch()` on the same expression is gated). - **Restriction-by-table is not analyzed.** A semijoin restriction such as `Ancestor & Undeclared` renders `Undeclared` as an `IN (subquery)` in the `WHERE` clause, which the gate's base-table analysis does not inspect — so an undeclared table used only as a restriction operand passes. (Joins *are* caught, because they extend the query's support; restrictions are not.) +- **Deletes are not gated.** `delete()` / `delete_quick()` inside `make()` are not intercepted by the runtime checks — a `make()` can delete rows from any table. The write checks cover `insert`/`insert1` and `update1` only. Comprehensive enforcement — verifying that a `make()` reads only from its declared ancestors and writes only to its target, across every access path — is fundamentally a **code-inspection problem, not a runtime-interception one.** That inspection is performed by the **DataJoint platform's code-deployment CI/CD process — not by the open-source core framework**. When pipeline code is deployed through the platform, its CI/CD **combines these runtime provenance checks with agentic review of the `make()` source** (backed by static analysis where it is sound) to cover the paths the runtime guardrail cannot — including the client-side gaps listed above (existence/count idioms, restriction-by-table, and access outside the Python client). The open-source `strict_provenance` guardrail complements that by catching accidental drift during local development; it does not replace it, and the core framework does not attempt code inspection itself. See [What is not in this specification](#what-is-not-in-this-specification). @@ -287,8 +293,12 @@ When `strict_provenance=True` and a `make()` is executing: | `self.insert1(row)` | **Allowed** (with key-consistency check, see below) | | `self.PartName.insert(rows)` | **Allowed** (with key-consistency check) | | `SomeOtherTable.insert(...)` | **Blocked** — raises `DataJointError`. | +| `self.update1(row)` | **Allowed** (with key-consistency check) | +| `SomeOtherTable.update1(...)` | **Blocked** — raises `DataJointError`. | | Reading from `self.upstream[Ancestor]` and writing to `Ancestor` | **Blocked** — `Ancestor` is not `self`. | +`update1` is gated like insert — the target must be `self` or one of its Parts, and the updated row's key columns must match the current `key`. The blocked-update error reads: `strict_provenance=True: update1 on '' is not permitted inside make() for ''. Only the target table and its Part tables may be written.` + As with reads, the write check is wired into the DataJoint insert path and so shares the same client-side limits described in [Enforcement model and its limits](#enforcement-model-and-its-limits). ### Key consistency @@ -302,6 +312,10 @@ This prevents a `make()` from inserting rows for entities it wasn't called with The key-consistency check applies to **dict-style rows**. Positional rows (tuples, numpy records) carry no attribute names to check against the current `key`, so they pass unchecked — dict inserts are the provenance-safe form under strict mode. +A second exception is server-side inserts: **`INSERT … SELECT` (inserting from a `QueryExpression`) runs entirely server-side; its rows never materialize client-side, so per-row key consistency is not applied on that path** (the target check still governs the destination). + +The same key-consistency rule governs updates. For `update1`, a mismatched key raises: `strict_provenance=True: updated row's ''= does not match the current make() key's ''=. Updates must be consistent with the key being populated.` + ### Default behavior `strict_provenance=False` (the default) leaves all `make()` behavior unchanged. No deprecation warning, no soft enforcement, no compatibility shim. Existing pipelines run identically.