docs: post-audit reconciliation — provenance limits, cascade U3/limitations, trace cross-refs

src/about/whats-new-23.md

@@ -8,7 +8,7 @@ DataJoint 2.3 introduces the **provenance trinity** — `Diagram.trace`, `self.u
 
 ## Overview
 
-DataJoint's provenance guarantee rests on the convention that `make(self, key)` reads only from declared upstream dependencies and writes only to `self` (and its Parts). The framework has always *defined* this boundary but never *checked* it: a `make()` could `fetch()` from any table, making the dependency invisible to the foreign-key graph and silently breaking the provenance claim downstream.
+DataJoint's provenance model rests on the convention that `make(self, key)` reads only from declared upstream dependencies and writes only to `self` (and its Parts). The framework has always *defined* this boundary but never *checked* it: a `make()` could `fetch()` from any table, making the dependency invisible to the foreign-key graph and silently breaking the provenance claim downstream.
 
 The 2.3 trinity closes that loop with three pieces designed as a unit:
 

src/how-to/run-computations.md

@@ -246,8 +246,48 @@ class FilteredComputation(dj.Computed):
 | `reserve_jobs` | `False` | Reserve jobs for distributed computing |
 | `suppress_errors` | `False` | Continue on errors |
 
+## Provenance-safe make() (strict provenance)
+
+A `make()` is provenance-safe when it reads only from its declared upstream dependencies and writes only to `self` (and its Parts). DataJoint 2.3 gives you an ergonomic read channel and an opt-in runtime guardrail to keep it that way.
+
+### Read ancestors through `self.upstream`
+
+Inside `make()`, `self.upstream[T]` returns the ancestor table `T` pre-restricted to the rows that contributed to the current `key`:
+
+```python
+def make(self, key):
+    # Reads pre-restricted to the current key — no manual `& key`
+    rate = self.upstream[Recording].fetch1('sampling_rate')
+    samples = self.upstream[Recording].to_arrays('signal')
+
+    result = compute_spectrum(samples, rate)
+    self.insert1({**key, 'spectrum': result})
+```
+
+`self.upstream` exposes declared ancestors only. To read your table's **own** Parts mid-`make()`, use `self.PartName` directly — that is permitted under strict mode.
+
+### Roll out `strict_provenance` in stages
+
+`dj.config["strict_provenance"]` defaults to `False`, so nothing changes until you opt in. Adopt it incrementally:
+
+1. **Enable it in staging** and run your pipeline:
+
+    ```python
+    dj.config["strict_provenance"] = True
+    ```
+
+2. **Fix the undeclared reads the guardrail surfaces.** Each violation raises a `DataJointError` naming the offending table. Either route the read through `self.upstream[T]` (if `T` is a declared ancestor) or declare the missing foreign-key dependency.
+3. **Enable it in production** once staging is clean.
+
+### What the guardrail does and does not catch
+
+When `strict_provenance=True`, DataJoint checks — through the Python client — that a `make()` reads only from declared ancestors and writes (`insert`/`insert1`/`update1`) only to `self` and its Parts, with each written row's key consistent with the current `key`.
+
+It is a **best-effort development guardrail, not an airtight boundary.** Raw SQL, other clients, existence/count idioms (`len(q)`, `bool(q)`, `key in q`), restriction-by-table, and `delete()` are **not** intercepted. See the [Provenance Spec — Enforcement model and its limits](../reference/specs/provenance.md#enforcement-model-and-its-limits) for the full list, and the [Provenance Specification](../reference/specs/provenance.md) for the migration path in detail.
+
 ## See Also
 
 - [Computation Model](../explanation/computation-model.md/) — How computation works
+- [Provenance Specification](../reference/specs/provenance.md) — `Diagram.trace`, `self.upstream`, `strict_provenance`
 - [Distributed Computing](distributed-computing.md) — Multi-worker setup
 - [Handle Errors](handle-errors.md) — Error recovery

src/reference/specs/autopopulate.md

@@ -317,7 +317,28 @@ def make(self, key):
 
 **Upstream-only convention:** Inside `make()`, fetch only from tables that are strictly upstream in the pipeline—tables referenced by foreign keys in the definition, their ancestors, and their part tables. This ensures reproducibility: computed results depend only on their declared dependencies.
 
-This convention is not currently enforced programmatically but is critical for pipeline integrity. Some pipelines violate this rule for operational reasons, which makes them non-reproducible. A future release may programmatically enforce upstream-only fetches inside `make()`.
+As of DataJoint 2.3, this convention is checked (opt-in) at runtime: setting `dj.config["strict_provenance"] = True` makes reads from undeclared tables and writes to tables other than `self` (and its Parts) raise `DataJointError` inside `make()`. The check is a best-effort development guardrail; see the [Provenance Specification](provenance.md) for the enforcement model and its documented limits.
+
+### 4.3.1 `self.upstream`: Pre-Restricted Ancestor Access
+
+Inside `make()`, the framework provides `self.upstream` — a pre-constructed `Diagram.trace(self & key)` — as the ergonomic way to read declared ancestors without manual `& key` restriction:
+
+```python
+def make(self, key):
+    # Each ancestor arrives pre-restricted to the rows that contributed to this key
+    rate = self.upstream[Recording].fetch1('sampling_rate')
+    trials = self.upstream[Trial].to_dicts()
+    self.insert1({**key, 'result': compute(trials, rate)})
+```
+
+Behavior:
+
+- **Lifecycle:** `self.upstream` is set to `Diagram.trace(self & key)` immediately before `make()` is invoked and cleared afterward — including when `make()` raises. Accessing it outside `make()` raises `DataJointError`.
+- **Lazy:** the trace diagram is built once per `make()` call; no SQL fires until an ancestor is accessed and fetched. Fetch results are not cached — reading the same ancestor twice issues two queries.
+- **Tripartite:** `self.upstream` is available across all tripartite phases (`make_fetch`, `make_compute`, `make_insert`) of the same `make()` call.
+- **Scope:** it exposes declared ancestors only. A table's own Parts are descendants, not ancestors — read them directly as `self.PartName`.
+
+See [Provenance Specification §2](provenance.md#2-selfupstream-inside-make) for the normative semantics.
 
 ### 4.4 Tripartite Make Pattern
 

src/reference/specs/cascade.md

@@ -33,7 +33,7 @@ When propagating a restriction across an edge `parent → child`, one of three r
 
 | Rule | Trigger | Effect on child |
 |---|---|---|
-| **F1. Copy** | `not aliased and parent_attrs ⊆ child_pk` | Child inherits the parent's restriction directly (same attribute names; literal restriction values copy as-is). |
+| **F1. Copy** | `not aliased and parent_attrs` **non-empty** `and parent_attrs ⊆ child_pk` | Child inherits the parent's restriction directly (same attribute names; literal restriction values copy as-is). An empty attribute set takes rule 3. |
 | **F2. Aliased rename** | `aliased` | Child's restriction is `parent.proj(**{fk_col: parent_col for fk_col, parent_col in attr_map.items()})` — the parent expression with columns renamed to match the child's column names. |
 | **F3. Project** | `not aliased and parent_attrs ⊄ child_pk` | Child's restriction is `parent.proj()` — the parent projected to its primary key. |
 
@@ -45,9 +45,9 @@ Symmetric inverses of the forward rules. Used by `part_integrity="cascade"` to p
 
 | Rule | Trigger | Effect on parent |
 |---|---|---|
-| **U1. Copy** | `not aliased and child_attrs ⊆ parent_pk` | Parent inherits the child's restriction directly (shared attribute names). |
+| **U1. Copy** | `not aliased and child_attrs` **non-empty** `and child_attrs ⊆ parent_pk` | Parent inherits the child's restriction directly (shared attribute names). An empty attribute set takes rule 3. |
 | **U2. Aliased reverse-rename** | `aliased` | Parent's restriction is `child.proj(**{parent_col: fk_col for fk_col, parent_col in attr_map.items()})` — the child expression with FK columns renamed back to the parent's column names. |
-| **U3. Project** | `not aliased and child_attrs ⊄ parent_pk` | Parent's restriction is `child.proj()` — the child projected to attributes in the parent's primary key. |
+| **U3. Project** | `not aliased and child_attrs ⊄ parent_pk` | Parent's restriction is `child.proj(*attr_map.keys())` — the child projected onto its FK columns (which, when non-aliased, share names with the parent's PK) so the parent restriction joins on the right columns. |
 
 ## Cascade flow
 
@@ -71,7 +71,7 @@ Master-Part integrity reflects the contract that a Master row exists for every P
 
 | Mode | Behavior |
 |---|---|
-| `"enforce"` (default) | If a delete would remove a Part row without removing the corresponding Master row, the entire delete is rolled back with `DataJointError`. The Master row is checked **after** the delete; the integrity violation is detected post-hoc and reversed. |
+| `"enforce"` (default) | If a delete would remove a Part row without removing the corresponding Master row, the entire delete is rolled back with `DataJointError`. The Master row is checked **after** the delete; the integrity violation is detected post-hoc and reversed. The *intent* is row-level (each deleted Part row should have its Master deleted), but the shipped post-check is **table-level** — see [Limitations](#limitations) for the resulting rare false negatives and false positives. |
 | `"ignore"` | No upward propagation; no post-check. Use when the Master row is intentionally preserved and the user has accepted that Part rows may be orphaned. Caller is responsible for the consequences. |
 | `"cascade"` | **Upward propagation enabled.** When cascade reaches a Part, the Master is also restricted (via the upward rules below), and the Master then forward-cascades back down to **all** its Parts (siblings of the originating Part included). Used when the user wants the master-part group treated atomically. |
 
@@ -145,7 +145,7 @@ When `(Subject.Recording & {"recording_id": 5}).delete(part_integrity="cascade")
 2. **FK path.** `shortest_path(Subject, Recording) = [Subject, Subject.Session, alias_node, Subject.Recording]` → real path `[Subject, Subject.Session, Subject.Recording]`.
 3. **Walk reversed.**
    - Edge `Subject.Session → Subject.Recording`: `aliased=True`. Apply **U2** — `Subject.Session` is restricted by `Subject.Recording.proj(subject_id='src_subject', session_id='src_session')`.
-   - Edge `Subject → Subject.Session`: `aliased=False`, `child_attrs={subject_id, session_id} ⊆ parent_pk={subject_id}`? No (`session_id` not in parent pk). Apply **U3** — `Subject` is restricted by `Subject.Session.proj()` (projected to `subject_id`).
+   - Edge `Subject → Subject.Session`: `aliased=False`, `child_attrs={subject_id, session_id} ⊆ parent_pk={subject_id}`? No (`session_id` not in parent pk). Apply **U3** — `Subject` is restricted by `Subject.Session.proj(*attr_map.keys())`, projecting the child onto its FK columns; for this primary FK those columns are just `subject_id`, so this is equivalent to `Subject.Session.proj()` projected to `subject_id`.
 4. **Materialize Master.** `Subject`'s restriction is fetched into a value tuple; replaces the chained `QueryExpression`.
 5. **Forward pass.** Master forward-cascades back down to `Subject.Session` and `Subject.Recording` (and any sibling Parts not on the original path), now with the materialized restriction.
 
@@ -191,10 +191,19 @@ For a cascade subgraph with N nodes and E edges, propagation runs in at most O(N
 
 ## What is not part of this specification
 
-- **`Diagram.trace()`** for general upstream restriction propagation: a related but distinct feature tracked in [#1423](https://github.com/datajoint/datajoint-python/issues/1423). `trace()` exposes upstream propagation as a first-class operator; the cascade engine's upward walk above is a closed implementation detail of `part_integrity="cascade"`.
+- **`Diagram.trace()`** for general upstream restriction propagation: a related but distinct feature that **shipped in 2.3** and reuses the same upward rules (U1/U2/U3) defined above. `trace()` exposes upstream propagation as a first-class operator; the cascade engine's upward walk in this document is the same machinery applied inside `part_integrity="cascade"`. See the [Provenance Specification](provenance.md) for `trace`'s API and semantics.
 - **Custom propagation rules** (user-defined): not supported. The three forward and three upward rules cover the cases the FK graph can produce.
 - **Cross-schema cascade**: handled by `dependencies.load_all_downstream()` called from `Diagram.cascade()`; orthogonal to the propagation rules described here.
 
+## Limitations
+
+The following are known, documented behaviors of the cascade engine as shipped:
+
+- **Single FK path (part→master walk).** The upward walk uses `nx.shortest_path` to find the FK chain from a Part to its Master. If a Part reaches its Master through multiple distinct FK chains, restrictions carried by the non-shortest paths are not applied. Workaround: use `part_integrity="ignore"` and perform the additional deletes manually.
+- **Materialization memory cost.** The master restriction is materialized via `to_arrays()` (to avoid MySQL error 1093, see [Materialization at the Master](#materialization-at-the-master)). The cost is bounded by the number of distinct master rows referenced by the matching parts. Cascade **preview** (`Diagram.cascade(...).counts()`) pays the same materialization cost as an actual delete.
+- **Empty-match sentinel.** When no master rows match, the master carries an always-false restriction and appears with zero rows in `counts()` and iteration. This is by design, not an error.
+- **Enforce granularity.** The `part_integrity="enforce"` post-check is **table-level**: it verifies that *some* rows of the master table were deleted whenever part rows were, not that each deleted part row's *specific* master row was deleted. As a result, rare false negatives (an unrelated master row happened to be deleted in the same cascade, masking a genuine orphan) and false positives (deleting already-orphaned part rows whose master was removed earlier) are possible.
+
 ## References
 
 - Source: `src/datajoint/diagram.py` — `Diagram.cascade`, `_propagate_restrictions`, `_apply_propagation_rule`, `_apply_propagation_rule_upward`, `_propagate_part_to_master`.

src/reference/specs/diagram.md

@@ -143,7 +143,7 @@ Create a cascade diagram for delete. Builds a complete dependency graph from the
 
 | Value | Behavior |
 |-------|----------|
-| `"enforce"` | Error if parts would be deleted before masters |
+| `"enforce"` | Default. The preview itself never errors; master-part integrity is enforced by `Table.delete()`'s post-check, which rolls back the delete if part rows would be removed without their masters (see [Cascade Spec](cascade.md#part_integrity-modes)). |
 | `"ignore"` | Allow deleting parts without masters |
 | `"cascade"` | Propagate restriction upward from part to master, then re-propagate downstream to all sibling parts |
 
@@ -154,6 +154,32 @@ With `"cascade"`, the restriction flows **upward** from a part table to its mast
 dj.Diagram.cascade(Session & {'subject_id': 'M001'}).counts()
 ```
 
+`part_integrity` accepts only `"enforce"`, `"ignore"`, or `"cascade"`; any other value raises `ValueError`.
+
+### `Diagram.trace()` (class method)
+
+```python
+dj.Diagram.trace(table_expr)
+```
+
+The **upstream mirror of `cascade()`**. Where `cascade()` walks downstream to every descendant, `trace()` walks **upstream** from a (possibly restricted) table expression to every ancestor across all loaded schemas, propagating the restriction along the way. Like `cascade()`, convergence is **OR** — an ancestor is included if reachable through *any* FK path from the seed.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `table_expr` | QueryExpression | — | A restricted table expression whose ancestry is traced |
+
+**Returns:** New `Diagram` containing the seed and its ancestors, each pre-restricted through the FK join path.
+
+Access a pre-restricted ancestor by indexing the trace:
+
+```python
+trace = dj.Diagram.trace(MyChild & key)
+session = trace[Session]              # Session restricted to ancestors of MyChild & key
+session.fetch1('session_date')
+```
+
+`trace()` reuses the same **upward propagation rules** (U1/U2/U3) documented in the [Cascade Spec](cascade.md#upward-propagation-child-parent). The [Provenance Specification §1](provenance.md#1-diagramtracetable_expr) is the normative spec for its API and semantics; inside `make()`, `self.upstream` is the per-`key` instance of a trace.
+
 ### `restrict()`
 
 ```python