docs: post-audit reconciliation — provenance limits, cascade U3/limitations, trace cross-refs#195
Open
dimitri-yatsenko wants to merge 1 commit into
Open
docs: post-audit reconciliation — provenance limits, cascade U3/limitations, trace cross-refs#195dimitri-yatsenko wants to merge 1 commit into
dimitri-yatsenko wants to merge 1 commit into
Conversation
…ations, trace cross-refs
Follow-up to the comprehensive 2.3 post-release audit of part integrity,
provenance, and cascade/trace/upstream across code and docs.
provenance.md:
- Enforcement limits: add ungated deletes; extend the existence/count-idiom
bullet to Aggregation/Union results.
- Write enforcement: update1 rows (gated like insert; exact error messages).
- Key consistency: INSERT ... SELECT server-side exception; update1 mismatch
message.
- §2: own Parts not reachable via self.upstream (read as self.PartName);
merge/master-part boundary note (trace does not descend into an ancestor
Master's Parts).
- Config wording: flag read from connection._config; soften 'provenance
guarantee' to 'provenance model'.
cascade.md:
- U3 corrected to child.proj(*attr_map.keys()) (and Worked Example 1 step 3).
- F1/U1 triggers: non-empty attribute set required; empty set takes rule 3.
- New Limitations section: single-FK-path upward walk, materialization memory
cost (preview pays it too), empty-match sentinel, enforce post-check
table-level granularity.
- Refresh stale trace paragraph: trace shipped in 2.3, reuses U1/U2/U3.
diagram.md:
- Add Diagram.trace() section cross-linking provenance.md §1.
- Correct 'enforce' preview claim (preview never errors; enforcement is
Table.delete()'s post-check); note part_integrity value validation.
autopopulate.md:
- De-stale §4.3 ('a future release may enforce' -> strict_provenance shipped
in 2.3, opt-in).
- New §4.3.1 documenting self.upstream (lifecycle, laziness, tripartite
availability, ancestors-only scope).
run-computations.md:
- New task-oriented section: provenance-safe make() — self.upstream usage,
staged strict_provenance rollout, what the guardrail does/doesn't catch.
whats-new-23.md: 'provenance guarantee' -> 'provenance model'.
dimitri-yatsenko
commented
Jul 6, 2026
| **Upstream-only convention:** Inside `make()`, fetch only from tables that are strictly upstream in the pipeline—tables referenced by foreign keys in the definition, their ancestors, and their part tables. This ensures reproducibility: computed results depend only on their declared dependencies. | ||
|
|
||
| This convention is not currently enforced programmatically but is critical for pipeline integrity. Some pipelines violate this rule for operational reasons, which makes them non-reproducible. A future release may programmatically enforce upstream-only fetches inside `make()`. | ||
| As of DataJoint 2.3, this convention is checked (opt-in) at runtime: setting `dj.config["strict_provenance"] = True` makes reads from undeclared tables and writes to tables other than `self` (and its Parts) raise `DataJointError` inside `make()`. The check is a best-effort development guardrail; see the [Provenance Specification](provenance.md) for the enforcement model and its documented limits. |
Member
Author
There was a problem hiding this comment.
Reading from any table that is not part of the "trace" (all tables above in the DAG and their part tables) will raise an error, not "undeclared tables."
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documentation batch from the comprehensive 2.3 post-release audit of part integrity, provenance, and cascade/trace/upstream (code vs docs). Six files.
Per-file summary
reference/specs/provenance.mddelete()/delete_quick()insidemake()); extended the existence/count-idiom bullet to noteAggregation/Unionlen/boolshare the bypass.update1rows — gated like insert (target must beself/Parts, key-consistent), with the exact blocked-update error message.INSERT … SELECTserver-side exception (per-row key check does not apply; target check still governs) and theupdate1mismatch message.self.upstream[...](read them asself.PartName), and added a merge/master-part boundary note — trace does not descend from an ancestor Master into its Parts (Parent → Master.Part → Master → Child: trace reachesMasteronly).connection._config); softened "provenance guarantee" → "provenance model".reference/specs/cascade.mdchild.proj(*attr_map.keys())(spec had the pre-fixchild.proj()form); Worked Example 1 step 3 updated.nx.shortest_path), materialization memory cost (preview pays it too), empty-match sentinel, and the enforce post-check's table-level granularity (rare false negatives/positives).reference/specs/diagram.md— added aDiagram.trace()section (was absent — split-brain with provenance.md) cross-linking the normative spec; corrected the claim that"enforce"makes preview error (preview never errors; enforcement isTable.delete()'s post-check); notedpart_integrityvalue validation.reference/specs/autopopulate.md— de-staled §4.3 ("a future release may enforce…" →strict_provenanceshipped in 2.3, opt-in) and added §4.3.1 documentingself.upstream(lifecycle incl. exception-path cleanup, outside-make()error, laziness/no caching, tripartite availability, ancestors-only scope).how-to/run-computations.md— new task-oriented section "Provenance-safe make()": reading ancestors viaself.upstream[T], stagedstrict_provenancerollout (staging → fix → production), and what the guardrail does/doesn't catch.about/whats-new-23.md— "provenance guarantee" → "provenance model".Companion code PRs (docs written to the post-fix state)
update1as gated, matching that fix.fix/cascade-multi-part-master(forthcoming) — fixes the once-per-master under-restriction inpart_integrity="cascade"and addspart_integrityvalue validation toDiagram.cascade(); accordingly this PR does not document the once-per-master behavior as a limitation, and diagram.md's validation note assumes it.If either companion PR changes shape in review, the corresponding lines here should be adjusted before merge.