Skip to content

docs(storage): settings-driven section prefixes, per-schema hash paths, GC semantics#196

Open
dimitri-yatsenko wants to merge 6 commits into
mainfrom
docs/storage-prefix-settings
Open

docs(storage): settings-driven section prefixes, per-schema hash paths, GC semantics#196
dimitri-yatsenko wants to merge 6 commits into
mainfrom
docs/storage-prefix-settings

Conversation

@dimitri-yatsenko

Copy link
Copy Markdown
Member

Documentation counterpart to datajoint/datajoint-python#1479 (in review), which aligned the code to the model these docs already described: the per-store hash_prefix / schema_prefix / filepath_prefix settings control the storage layout and are consumed by writers, garbage collection, and <filepath@> validation from the same spec values (no component-level fallbacks).

The load-bearing point (per maintainer review)

Hash-addressed paths embed the schema name{hash_prefix}/{schema}/{hash} — which:

  • scopes deduplication per schema (the type-system spec previously claimed "across the entire project" — corrected);
  • lets GC attribute every stored object to a schema;
  • and, because storage-side listing covers the whole section, means collect() must be given every schema that uses the store — the GC how-to's old note ("you only need to scan schemas that share the same database") asserted the opposite and is replaced with an explicit warning. Per-schema attribution is also what makes dropped-schema debris reclaimable.

Other corrections

  • Layout parameterization: all path illustrations now show the configurable sections; schema-addressed pages no longer show the root-level {schema}/… layout (that's the ≤2.3.0 legacy layout, noted as such; GC lists both).
  • Write tokens: schema-addressed filenames are {field}_{token}{ext} — several pages showed a dot-separated {field}.{token}.{ext} form that never matched the code.
  • GC how-to: token-per-write versioning, folder objects + .manifest.json sidecars with coverage-based orphan matching, and the filepath_prefix never-touched guarantee.
  • object-store-configuration spec: new normative Consumption paragraph (single source of values, defaults applied to every spec including plugin protocols), the hash-paths-embed-schema note, and the changing-prefixes-on-populated-store caveat (also added as a warning in configure-storage).

Sequencing

Merge after datajoint/datajoint-python#1479 — these pages describe the post-fix behavior. No file overlap with the open #195.

…s, GC semantics

Reflects the storage corrections in datajoint/datajoint-python#1479, which
aligned the code to these docs' long-documented model: the per-store
hash_prefix/schema_prefix/filepath_prefix settings control the layout and are
consumed by writers, garbage collection, and <filepath@> validation alike.

Emphasis per review: hash-addressed paths embed the schema name
({hash_prefix}/{schema}/{hash}), which scopes deduplication per schema and is
what garbage collection relies on to attribute stored objects to schemas —
and, because storage-side listing covers the whole section, collect() must be
given every schema that uses the store. The GC how-to's old 'you only need to
scan schemas that share the same database' note said the opposite and is
replaced with a warning stating the real obligation.

- how-to/garbage-collection.md: corrected multi-schema warning; layout blocks
  parameterized by the prefix settings; token-per-write versioning, folder
  objects + manifest sidecars and coverage-based orphan matching; legacy
  root-level layout note (2.3.0 and earlier); filepath_prefix never touched.
- reference/specs/object-store-configuration.md: normative Consumption
  paragraph (all components read the same spec values; defaults applied to
  every spec incl. plugin protocols; no component-level fallbacks);
  hash-paths-embed-schema note; changing-prefixes-on-existing-store caveat.
- reference/specs/type-system.md: per-project dedup claim corrected to
  per-schema; storage tree and codec table paths sectioned and parameterized
  ({hash_prefix}/{schema}/{hash}; {schema_prefix}/{schema}/{table}/...).
- explanation/type-system.md, how-to/use-object-storage.md,
  how-to/manage-pipeline-project.md, reference/specs/staged-insert.md,
  reference/specs/npy-codec.md: root-level schema-addressed paths corrected
  to the sectioned layout with write tokens.
- reference/configuration.md, how-to/configure-storage.md,
  how-to/choose-storage-type.md, object-store-configuration.md: token
  filename format corrected to {field}_{token}{ext} (docs showed dot-
  separated); configure-storage gains the changing-prefixes warning and the
  single-source consumption note.
…helper

format_stats was removed from datajoint-python (it presumed one display format
and the library never called it). Examples now read the plain dict scan()/
collect() return, and note that collect(dry_run=True) reports counts only —
use scan() for per-item paths and reclaimable bytes.
@dimitri-yatsenko

Copy link
Copy Markdown
Member Author

Added a commit removing dj.gc.format_stats(stats) from the examples (garbage-collection.md, manage-large-data.md). That helper is being removed in datajoint/datajoint-python#1479 — it presumed a single display format and the library never called it. Examples now inspect the plain stats dict directly, and note that collect(dry_run=True) reports counts only (use scan() for per-item paths and reclaimable bytes).

…ombined totals

Reflects the GC refactor in datajoint/datajoint-python#1479: garbage collection
is a store-bound GarbageCollector class, not module-level functions.

- All examples use dj.gc.GarbageCollector(store=...).scan(...) / .collect(...).
- The "Multiple Schemas" section is inverted: per-schema scoping makes any
  SUBSET of a store's schemas safe to collect — the old "always pass every
  schema or lose data" warning was the opposite of the shipped behavior and is
  replaced with a per-schema-attribution note.
- Named-store examples bind the store at construction (no store_name= param).
- Combined totals (orphaned/orphaned_bytes/referenced/stored) removed from the
  statistics reference and examples; readers sum the per-section fields.
@dimitri-yatsenko

Copy link
Copy Markdown
Member Author

Added a commit updating the GC docs to the store-bound GarbageCollector class (datajoint/datajoint-python#1479): examples use dj.gc.GarbageCollector(store=...).scan(...)/.collect(...); the store binds at construction (no store_name= param). Two notable content changes: the Multiple Schemas guidance is now inverted — per-schema scoping makes any subset safe, so the old "always pass every schema or lose data" warning is replaced with a per-schema-attribution note; and the removed combined totals (orphaned/orphaned_bytes/referenced/stored) are dropped from the statistics reference (readers sum the per-section fields).

…n) replaces scan

Follows datajoint/datajoint-python#1479: the collector takes its schemas at
construction (dj.gc.GarbageCollector(schema1, schema2, store=...)), and the
separate scan() is gone — collect(dry_run=True) is the default read-only
report (full orphan paths + reclaimable bytes), collect(dry_run=False) deletes.
…h_paths)

Mirror the schema-addressed naming on the hash side of the collect() report
(datajoint/datajoint-python#1479): hash_referenced->hash_paths_referenced,
hash_stored->hash_paths_stored, hash_orphaned->hash_paths_orphaned,
hash_orphaned_bytes->hash_paths_orphaned_bytes, orphaned_hashes->orphaned_hash_paths.
- type-system.md <hash@>: fix stale intro path to {hash_prefix}/{schema}/{hash}
  matching the tree diagram and codec table
- type-system.md <hash@>: correct the two bullets that still claimed per-project
  scope / cross-schema sharing to per-schema, consistent with the Deduplication bullet
- npy-codec.md: update the example and JSON metadata paths to the
  {schema_prefix}/.../{attribute}_{token}.npy form used elsewhere
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant