From 9d6e40fbcbea5db1042255ee0d9e568f9b1ffa01 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 6 Jul 2026 13:10:58 -0500 Subject: [PATCH 1/6] docs(storage): settings-driven section prefixes, per-schema hash paths, GC semantics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reflects the storage corrections in datajoint/datajoint-python#1479, which aligned the code to these docs' long-documented model: the per-store hash_prefix/schema_prefix/filepath_prefix settings control the layout and are consumed by writers, garbage collection, and validation alike. Emphasis per review: hash-addressed paths embed the schema name ({hash_prefix}/{schema}/{hash}), which scopes deduplication per schema and is what garbage collection relies on to attribute stored objects to schemas — and, because storage-side listing covers the whole section, collect() must be given every schema that uses the store. The GC how-to's old 'you only need to scan schemas that share the same database' note said the opposite and is replaced with a warning stating the real obligation. - how-to/garbage-collection.md: corrected multi-schema warning; layout blocks parameterized by the prefix settings; token-per-write versioning, folder objects + manifest sidecars and coverage-based orphan matching; legacy root-level layout note (2.3.0 and earlier); filepath_prefix never touched. - reference/specs/object-store-configuration.md: normative Consumption paragraph (all components read the same spec values; defaults applied to every spec incl. plugin protocols; no component-level fallbacks); hash-paths-embed-schema note; changing-prefixes-on-existing-store caveat. - reference/specs/type-system.md: per-project dedup claim corrected to per-schema; storage tree and codec table paths sectioned and parameterized ({hash_prefix}/{schema}/{hash}; {schema_prefix}/{schema}/{table}/...). - explanation/type-system.md, how-to/use-object-storage.md, how-to/manage-pipeline-project.md, reference/specs/staged-insert.md, reference/specs/npy-codec.md: root-level schema-addressed paths corrected to the sectioned layout with write tokens. - reference/configuration.md, how-to/configure-storage.md, how-to/choose-storage-type.md, object-store-configuration.md: token filename format corrected to {field}_{token}{ext} (docs showed dot- separated); configure-storage gains the changing-prefixes warning and the single-source consumption note. --- src/explanation/type-system.md | 8 +-- src/how-to/choose-storage-type.md | 2 +- src/how-to/configure-storage.md | 11 +++- src/how-to/garbage-collection.md | 53 +++++++++++++------ src/how-to/manage-pipeline-project.md | 2 +- src/how-to/use-object-storage.md | 2 +- src/reference/configuration.md | 6 +-- src/reference/specs/npy-codec.md | 10 ++-- .../specs/object-store-configuration.md | 29 +++++++++- src/reference/specs/staged-insert.md | 4 +- src/reference/specs/type-system.md | 34 +++++++----- 11 files changed, 110 insertions(+), 51 deletions(-) diff --git a/src/explanation/type-system.md b/src/explanation/type-system.md index a6a32b80..53acfa12 100644 --- a/src/explanation/type-system.md +++ b/src/explanation/type-system.md @@ -266,14 +266,14 @@ result = np.mean(ref) # Downloads automatically ### `` — Schema-Addressed Storage -Schema-addressed storage for files and folders. Path mirrors the database structure: `{schema}/{table}/{pk}/{attribute}`. +Schema-addressed storage for files and folders. Path mirrors the database structure within the store's schema-addressed section: `{schema_prefix}/{schema}/{table}/{pk}/{attribute}_{token}` (`schema_prefix` defaults to `_schema`). ```python class RecordingAnalysis(dj.Computed): definition = """ -> Recording --- - results : # Stored at {schema}/{table}/{pk}/results/ + results : # Stored at _schema/{schema}/{table}/{pk}/results_{token}/ """ ``` @@ -296,9 +296,9 @@ class RawData(dj.Manual): Object store codecs use one of two addressing schemes: -**Hash-addressed** — Path derived from content hash (e.g., `_hash/ab/cd/abcd1234...`). Provides automatic deduplication—identical content stored once. Used by ``, ``, ``. +**Hash-addressed** — Path derived from content hash, with the schema name embedded (e.g., `_hash/{schema}/ab/cd/abcd1234...`; section configurable via `hash_prefix`). Provides automatic deduplication—identical content stored once **per schema**. Used by ``, ``, ``. -**Schema-addressed** — Path mirrors database structure: `{schema}/{table}/{pk}/{attribute}`. Human-readable, browsable paths that reflect your data organization. No deduplication. Used by ``, ``, and plugin codecs (``, ``, ``). +**Schema-addressed** — Path mirrors database structure: `_schema/{schema}/{table}/{pk}/{attribute}_{token}` (section configurable via `schema_prefix`). Human-readable, browsable paths that reflect your data organization. No deduplication. Used by ``, ``, and plugin codecs (``, ``, ``). | Mode | Database | Object Store | Deduplication | Use Case | |------|----------|--------------|---------------|----------| diff --git a/src/how-to/choose-storage-type.md b/src/how-to/choose-storage-type.md index 5d579b7d..233e0ad3 100644 --- a/src/how-to/choose-storage-type.md +++ b/src/how-to/choose-storage-type.md @@ -396,7 +396,7 @@ file : # File attachments ### Schema-Addressed: `` or `` -**Storage:** Object store at `{store}/_schema/{schema}/{table}/{key}/{field}.{token}.ext` +**Storage:** Object store at `{store}/_schema/{schema}/{table}/{key}/{field}_{token}.ext` **Syntax:** ```python diff --git a/src/how-to/configure-storage.md b/src/how-to/configure-storage.md index b472bcb1..ba0fc51f 100644 --- a/src/how-to/configure-storage.md +++ b/src/how-to/configure-storage.md @@ -235,7 +235,7 @@ _hash/{schema}/ab/cd/abcdefghijklmnopqrstuvwxyz Schema-addressed storage (``, ``) does not use subfolding—it uses key-based paths: ``` -{location}/_schema/{partition}/{schema}/{table}/{key}/{field_name}.{token}.{ext} +{location}/_schema/{partition}/{schema}/{table}/{key}/{field_name}_{token}{ext} ``` ### Filesystem Recommendations @@ -282,7 +282,7 @@ This unified approach enables: ## Customizing Storage Sections -Each store is divided into sections for different storage types. By default, DataJoint uses `_hash/` for hash-addressed storage and `_schema/` for schema-addressed storage. You can customize the path prefix for each section using the `*_prefix` configuration parameters to map DataJoint to existing storage layouts: +Each store is divided into sections for different storage types. By default, DataJoint uses `_hash/` for hash-addressed storage and `_schema/` for schema-addressed storage. You can customize the path prefix for each section using the `*_prefix` configuration parameters to map DataJoint to existing storage layouts. Writers, garbage collection, and `` validation all read the same per-store values, so a relocated section stays consistent end to end: ```json { @@ -304,6 +304,13 @@ Each store is divided into sections for different storage types. By default, Dat - The `hash_prefix` and `schema_prefix` sections are reserved for DataJoint-managed storage - The `filepath_prefix` is optional (`null` = unrestricted, or set a required prefix) +!!! warning "Changing prefixes on an existing store" + Set the prefixes when the store is first configured. Changing them on a + store that already holds data leaves existing objects readable (their + paths are recorded in each row's metadata), but garbage collection scans + only the currently configured sections, and hash deduplication will not + match content stored under an old prefix. + **Example with hierarchical layout:** ```json diff --git a/src/how-to/garbage-collection.md b/src/how-to/garbage-collection.md index 6d2822e9..5dee6cda 100644 --- a/src/how-to/garbage-collection.md +++ b/src/how-to/garbage-collection.md @@ -79,10 +79,16 @@ stats = dj.gc.collect( ) ``` -!!! note "Per-schema deduplication" - Hash-addressed storage is deduplicated **within** each schema. Different - schemas have independent storage, so you only need to scan schemas that - share the same database. +!!! warning "Per-schema paths and multi-schema stores" + Every hash-addressed path embeds the schema name + (`{hash_prefix}/{schema}/{hash}`), so deduplication is scoped **within** + each schema and every stored object is attributable to the schema that + wrote it. Orphan detection, however, lists the **entire store** on the + storage side: a schema that uses the store but is not passed to + `scan()`/`collect()` has its live objects misclassified as orphans — + and deleted. Always pass **every** schema that uses the store. The same + per-schema attribution is what makes leftovers of fully dropped schemas + identifiable and reclaimable by a later `collect()`. ## Named Stores @@ -160,32 +166,47 @@ DataJoint uses two storage patterns: ### Hash-Addressed (``, ``, ``) ``` -_hash/ +{hash_prefix}/ # store setting, default: _hash/ {schema}/ ab/ cd/ abcdefghij... # Content identified by Base32-encoded MD5 hash ``` -- Duplicate content shares storage within each schema +- The section location comes from the store's `hash_prefix` setting + (default `_hash`) — the same value the writer uses, so scanner and writer + cannot drift +- Duplicate content shares storage within each schema — the schema name is + part of every path, which is what scopes deduplication and lets GC + attribute each stored object to a schema - Paths are stored in metadata—safe from config changes - Cannot delete until no rows reference the content -- GC compares stored paths against filesystem +- GC compares stored paths against the filesystem ### Schema-Addressed (``, ``) ``` -myschema/ - mytable/ - primary_key_values/ - attribute_name/ - data.zarr/ - data.npy +{schema_prefix}/ # store setting, default: _schema/ + {schema}/ + {table}/ + {primary_key_values}/ + {attribute}_{token}.npy # single-file object + {attribute}_{token}.zarr/ # folder object (many files) + {attribute}_{token}.zarr.manifest.json ``` -- Each row has unique path based on schema structure -- Paths mirror database organization -- GC removes paths not referenced by any row +- The section location comes from the store's `schema_prefix` setting + (default `_schema`). DataJoint 2.3.0 and earlier wrote these objects at + root level (`{schema}/...`); GC lists both layouts, so legacy objects + remain reclaimable +- Every write gets a unique random token, so multiple versions of a row's + object can coexist; GC reclaims superseded tokens +- Orphan matching is coverage-based: a stored file is live if it **is** a + referenced path, lies **under** a referenced folder object, or is its + `.manifest.json` sidecar — live folder objects are never partially + collected +- GC never touches the store's declared `filepath_prefix` namespace + (user-managed `` files) ## Troubleshooting diff --git a/src/how-to/manage-pipeline-project.md b/src/how-to/manage-pipeline-project.md index 7a90299d..0ccf3966 100644 --- a/src/how-to/manage-pipeline-project.md +++ b/src/how-to/manage-pipeline-project.md @@ -276,7 +276,7 @@ A DataJoint project creates a structured storage pattern: ### Considerations -- Object paths include schema name: `{project}/{schema}/{table}/...` +- Object paths include the schema name: `{project}/{schema_prefix}/{schema}/{table}/...` — the schema is embedded in every managed path (both sections), which scopes hash deduplication per schema and lets garbage collection attribute stored objects to schemas - Users need read access to fetch blobs from upstream schemas - Content-addressed storage (``) shares objects across tables - Garbage collection requires coordinated delete permissions diff --git a/src/how-to/use-object-storage.md b/src/how-to/use-object-storage.md index d3753462..fe0d0b77 100644 --- a/src/how-to/use-object-storage.md +++ b/src/how-to/use-object-storage.md @@ -134,7 +134,7 @@ Table.insert1({'id': 2, 'array': data}) # References same object `` and `` use **schema-addressed** storage: -- Objects stored at paths that mirror database schema: `{schema}/{table}/{pk}/{attribute}.npy` +- Objects stored under the store's schema-addressed section at paths that mirror the database schema: `_schema/{schema}/{table}/{pk}/{attribute}_{token}.npy` (section configurable via `schema_prefix`) - Browsable organization in object storage - One object per entity (no deduplication) - Supports lazy loading with metadata access diff --git a/src/reference/configuration.md b/src/reference/configuration.md index 022b1310..a1a08f03 100644 --- a/src/reference/configuration.md +++ b/src/reference/configuration.md @@ -117,7 +117,7 @@ Prefixes must be mutually exclusive (no prefix can be a parent/child of another) **How storage methods use stores:** - **Hash-addressed** (``, ``): `{location}/{hash_prefix}/{schema}/{hash}` with optional subfolding -- **Schema-addressed** (``, ``): `{location}/{schema_prefix}/{partition}/{schema}/{table}/{key}/{field}.{token}.{ext}` with optional partitioning +- **Schema-addressed** (``, ``): `{location}/{schema_prefix}/{partition}/{schema}/{table}/{key}/{field}_{token}{ext}` with optional partitioning - **Filepath** (``): `{location}/{filepath_prefix}/{user_path}` (user-managed, cannot use hash or schema prefixes) All storage methods share the same stores and default store. DataJoint reserves the configured `hash_prefix` and `schema_prefix` sections for managed storage; `` references can use any other paths (unless `filepath_prefix` is configured to restrict them). @@ -127,12 +127,12 @@ All storage methods share the same stores and default store. DataJoint reserves Without partitioning: ``` {location}/_hash/{schema}/ab/cd/abcd1234... # hash-addressed with subfolding -{location}/_schema/{schema}/{table}/{key}/data.x8f2a9b1.zarr # schema-addressed, no partitioning +{location}/_schema/{schema}/{table}/{key}/data_x8f2a9b1.zarr # schema-addressed, no partitioning ``` With `partition_pattern: "subject_id/session_date"`: ``` -{location}/_schema/subject_id=042/session_date=2024-01-15/{schema}/{table}/{remaining_key}/data.x8f2a9b1.zarr +{location}/_schema/subject_id=042/session_date=2024-01-15/{schema}/{table}/{remaining_key}/data_x8f2a9b1.zarr ``` If table lacks partition attributes, it follows normal path structure. diff --git a/src/reference/specs/npy-codec.md b/src/reference/specs/npy-codec.md index f6a44892..7882f984 100644 --- a/src/reference/specs/npy-codec.md +++ b/src/reference/specs/npy-codec.md @@ -12,7 +12,7 @@ and transparent numpy integration via the `__array__` protocol. **Key characteristics:** - **Store only**: Requires `@` modifier (`` or ``) -- **Schema-addressed**: Paths mirror database structure (`{schema}/{table}/{pk}/{attr}.npy`) +- **Schema-addressed**: Paths mirror database structure within the store's schema section (`{schema_prefix}/{schema}/{table}/{pk}/{attr}_{token}.npy`; `schema_prefix` defaults to `_schema`) - **Lazy loading**: Shape/dtype available without download - **Transparent**: Use directly in numpy operations - **Portable**: Standard `.npy` format readable by numpy, MATLAB, etc. @@ -68,7 +68,7 @@ ref.dtype # numpy.dtype: float64 ref.ndim # int: 2 ref.size # int: 32000 ref.nbytes # int: 256000 (estimated) -ref.path # str: "my_schema/recording/recording_id=1/waveform.npy" +ref.path # str: "_schema/my_schema/recording/recording_id=1/waveform_x8f2a9b1.npy" ref.store # str or None: store name ref.is_loaded # bool: False (until loaded) ``` @@ -189,7 +189,7 @@ storage that reflects your data model. ### Schema-Addressed Path Construction ``` -{schema}/{table}/{primary_key_values}/{attribute}.npy +{schema_prefix}/{schema}/{table}/{primary_key_values}/{attribute}_{token}.npy ``` Example: `lab_ephys/recording/recording_id=1/waveform.npy` @@ -237,10 +237,10 @@ Files are stored at predictable paths and can be accessed directly: ```python # Get the storage path ref = (Recording & 'recording_id=1').fetch1('waveform') -print(ref.path) # "my_schema/recording/recording_id=1/waveform.npy" +print(ref.path) # "_schema/my_schema/recording/recording_id=1/waveform_x8f2a9b1.npy" # Load directly with numpy (if you have store access) -arr = np.load('/path/to/store/my_schema/recording/recording_id=1/waveform.npy') +arr = np.load('/path/to/store/_schema/my_schema/recording/recording_id=1/waveform_x8f2a9b1.npy') ``` ## Comparison with Other Codecs diff --git a/src/reference/specs/object-store-configuration.md b/src/reference/specs/object-store-configuration.md index bfc23750..6fccd5b6 100644 --- a/src/reference/specs/object-store-configuration.md +++ b/src/reference/specs/object-store-configuration.md @@ -146,6 +146,31 @@ Each store is divided into sections controlled by prefix configuration. The `*_p - `null` (default): filepaths can use any path except reserved sections - `"some/prefix"`: all filepaths must start with this prefix +**Consumption (normative):** the prefix values are read from the store spec +by every component that touches managed storage — the writers (`put_hash`, +`build_object_path`), garbage collection (which scans the configured +`hash_prefix` section and never enters the configured `filepath_prefix` +namespace), and `` validation. Defaults are applied to **every** +store spec, including plugin protocols, when the spec is resolved +(`get_store_spec`), and there are no component-level fallbacks — all +components always observe the same values. + +**Hash paths embed the schema:** `{hash_prefix}/{schema}/{hash}`. +Deduplication is therefore scoped **per schema**, and every stored object is +attributable to the schema that wrote it. Garbage collection depends on this +attribution; because storage-side listing covers the whole section across +all schemas, a `collect()` must be given every schema that uses the store +(see [Clean Up Object Storage](../../how-to/garbage-collection.md)). + +**Changing prefixes on a store that already holds data** is not recommended: +existing objects remain readable (each row's metadata records its full +path), but garbage collection scans only the currently configured sections — +objects under a previous prefix are not reclamation candidates until the +setting is restored — and hash deduplication will not match content stored +under the old prefix. Schema-addressed objects written by DataJoint 2.3.0 +and earlier live at root-level `{schema}/...` paths; garbage collection +lists both layouts. + **Example with custom prefixes:** ```json @@ -362,12 +387,12 @@ Experiment.insert1({'experiment_id': 1, 'data': my_data}) **Path structure (no partitioning):** ``` -{location}/{schema_prefix}/{schema_name}/{table_name}/{key_string}/{field_name}.{token}.{ext} +{location}/{schema_prefix}/{schema_name}/{table_name}/{key_string}/{field_name}_{token}{ext} ``` **With partitioning:** ``` -{location}/{schema_prefix}/{partition_path}/{schema_name}/{table_name}/{remaining_key}/{field_name}.{token}.{ext} +{location}/{schema_prefix}/{partition_path}/{schema_name}/{table_name}/{remaining_key}/{field_name}_{token}{ext} ``` **Algorithm:** diff --git a/src/reference/specs/staged-insert.md b/src/reference/specs/staged-insert.md index d2295152..f8006e9b 100644 --- a/src/reference/specs/staged-insert.md +++ b/src/reference/specs/staged-insert.md @@ -55,10 +55,10 @@ The primary key must be fully set on `staged.rec` before `staged.store()` or `st Staged objects are written at the canonical schema-addressed path from the first byte. There is no intermediate staging location: the object exists at its final path during the write and is deleted from there if the block exits with an exception. ``` -{location}/{schema}/{table}/{pk_serialized}/{field}_{token}{ext} +{location}/{schema_prefix}/{schema}/{table}/{pk_serialized}/{field}_{token}{ext} ``` -Built by `storage.build_object_path`. `{location}` is the configured store base, `{token}` is a random suffix of `token_length` characters (default 8, per the store spec), `{pk_serialized}` is the serialized primary key, and partitioning follows the store's `partition_pattern`. +Built by `storage.build_object_path`. `{location}` is the configured store base, `{schema_prefix}` is the store's schema-addressed section (default `_schema`), `{token}` is a random suffix of `token_length` characters (default 8, per the store spec), `{pk_serialized}` is the serialized primary key, and partitioning follows the store's `partition_pattern`. ## Metadata contract diff --git a/src/reference/specs/type-system.md b/src/reference/specs/type-system.md index 994da1da..a379e1ae 100644 --- a/src/reference/specs/type-system.md +++ b/src/reference/specs/type-system.md @@ -246,7 +246,7 @@ returns the appropriate dtype based on storage mode: Schema-addressed OAS storage for complex, multi-part objects (files, folders, Zarr arrays, HDF5): -- **Schema-addressed**: Path mirrors database structure: `{schema}/{table}/{pk}/{attribute}/` +- **Schema-addressed**: Path mirrors database structure within the store's schema section: `{schema_prefix}/{schema}/{table}/{pk}/{attribute}_{token}` (`schema_prefix` defaults to `_schema`) - **Complex objects**: Can store directory structures with multiple files (e.g., Zarr arrays) - One-to-one relationship with table row - Deleted when row is deleted @@ -297,18 +297,24 @@ Hash-addressed storage with deduplication for individual, atomic objects: - **Per-project scope**: content is shared across all schemas in a project (not per-schema) - Many-to-one: multiple rows (even across schemas) can reference same content - Reference counted for garbage collection -- Deduplication: identical content stored once across the entire project +- Deduplication: identical content stored once **per schema** — the schema name is part of every hash path, which scopes deduplication and lets garbage collection attribute each stored object to its schema - **dtype**: `json` (stores hash, store name, size, metadata) ``` store_root/ -├── {schema}/{table}/{pk}/ # schema-addressed storage -│ └── {attribute}/ +├── {schema_prefix}/ # default _schema/ — schema-addressed storage +│ └── {schema}/{table}/{pk}/ +│ └── {attribute}_{token}[.ext] │ -└── _hash/ # hash-addressed storage - └── {hash[:2]}/{hash[2:4]}/{hash} +└── {hash_prefix}/ # default _hash/ — hash-addressed storage + └── {schema}/ + └── {hash[:2]}/{hash[2:4]}/{hash} ``` +Section prefixes come from the store settings (`hash_prefix`, `schema_prefix`; +see [Object Store Configuration](object-store-configuration.md#section-prefixes)). +Both sections embed the schema name in every path. + #### Implementation ```python @@ -593,15 +599,15 @@ Users can define custom codecs for domain-specific data. See the [Codec API Spec | Type | get_dtype | Resolves To | Storage Location | Dedup | Returns | |------|-----------|-------------|------------------|-------|---------| | `` | `bytes` | `LONGBLOB`/`BYTEA` | Database | No | Python object | -| `` | `` | `json` | `_hash/{hash}` | Yes | Python object | -| `` | `` | `json` | `_hash/{hash}` | Yes | Python object | +| `` | `` | `json` | `{hash_prefix}/{schema}/{hash}` | Yes | Python object | +| `` | `` | `json` | `{hash_prefix}/{schema}/{hash}` | Yes | Python object | | `` | `bytes` | `LONGBLOB`/`BYTEA` | Database | No | Local file path | -| `` | `` | `json` | `_hash/{hash}` | Yes | Local file path | -| `` | `` | `json` | `_hash/{hash}` | Yes | Local file path | -| `` | `json` | `JSON`/`JSONB` | `{schema}/{table}/{pk}/` | No | ObjectRef | -| `` | `json` | `JSON`/`JSONB` | `{schema}/{table}/{pk}/` | No | ObjectRef | -| `` | `json` | `JSON`/`JSONB` | `_hash/{hash}` | Yes | bytes | -| `` | `json` | `JSON`/`JSONB` | `_hash/{hash}` | Yes | bytes | +| `` | `` | `json` | `{hash_prefix}/{schema}/{hash}` | Yes | Local file path | +| `` | `` | `json` | `{hash_prefix}/{schema}/{hash}` | Yes | Local file path | +| `` | `json` | `JSON`/`JSONB` | `{schema_prefix}/{schema}/{table}/{pk}/` | No | ObjectRef | +| `` | `json` | `JSON`/`JSONB` | `{schema_prefix}/{schema}/{table}/{pk}/` | No | ObjectRef | +| `` | `json` | `JSON`/`JSONB` | `{hash_prefix}/{schema}/{hash}` | Yes | bytes | +| `` | `json` | `JSON`/`JSONB` | `{hash_prefix}/{schema}/{hash}` | Yes | bytes | | `` | `json` | `JSON`/`JSONB` | Configured store | No | ObjectRef | ## Garbage Collection for Hash Storage From 2a16f5fd0411dcec35ba512b12c0355c5430bcf5 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 6 Jul 2026 14:02:26 -0500 Subject: [PATCH 2/6] docs(gc): inspect the stats dict directly; drop removed format_stats helper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit format_stats was removed from datajoint-python (it presumed one display format and the library never called it). Examples now read the plain dict scan()/ collect() return, and note that collect(dry_run=True) reports counts only — use scan() for per-item paths and reclaimable bytes. --- src/how-to/garbage-collection.md | 12 +++++++----- src/how-to/manage-large-data.md | 4 ++-- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/src/how-to/garbage-collection.md b/src/how-to/garbage-collection.md index 5dee6cda..c5ca416f 100644 --- a/src/how-to/garbage-collection.md +++ b/src/how-to/garbage-collection.md @@ -30,13 +30,13 @@ Run garbage collection periodically to reclaim storage space. ```python import datajoint as dj -# Scan for orphaned items (dry run) +# Scan for orphaned items (read-only) stats = dj.gc.scan(schema1, schema2) -print(dj.gc.format_stats(stats)) +print(f"{stats['orphaned']} orphaned, {stats['orphaned_bytes'] / 1e6:.1f} MB reclaimable") # Remove orphaned items stats = dj.gc.collect(schema1, schema2, dry_run=False) -print(dj.gc.format_stats(stats)) +print(f"Deleted {stats['deleted']} items, freed {stats['bytes_freed'] / 1e6:.1f} MB") ``` ## Scan Before Collecting @@ -57,9 +57,11 @@ print(f"Total bytes: {stats['orphaned_bytes'] / 1e6:.1f} MB") The default `dry_run=True` reports what would be deleted without deleting: ```python -# Safe: shows what would be deleted +# Safe: reports how many items would be deleted (deletes nothing). +# For the per-item paths and reclaimable bytes, use dj.gc.scan() instead — +# collect(dry_run=True) reports bytes_freed as 0 because nothing was removed. stats = dj.gc.collect(my_schema, dry_run=True) -print(dj.gc.format_stats(stats)) +print(f"{stats['orphaned']} items would be deleted") # After review, actually delete stats = dj.gc.collect(my_schema, dry_run=False) diff --git a/src/how-to/manage-large-data.md b/src/how-to/manage-large-data.md index 18ca9777..555f5778 100644 --- a/src/how-to/manage-large-data.md +++ b/src/how-to/manage-large-data.md @@ -115,9 +115,9 @@ import datajoint as dj # Objects are NOT automatically deleted with rows (MyTable & old_data).delete() -# Scan for orphaned items +# Scan for orphaned items (read-only) stats = dj.gc.scan(my_schema) -print(dj.gc.format_stats(stats)) +print(f"{stats['orphaned']} orphaned, {stats['orphaned_bytes'] / 1e6:.1f} MB reclaimable") # Remove orphaned items stats = dj.gc.collect(my_schema, dry_run=False) From 17a6da7ea51877e5721e442a85cdd03c6c72d0d9 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 6 Jul 2026 17:06:47 -0500 Subject: [PATCH 3/6] docs(gc): store-bound GarbageCollector API; per-schema safety; drop combined totals MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reflects the GC refactor in datajoint/datajoint-python#1479: garbage collection is a store-bound GarbageCollector class, not module-level functions. - All examples use dj.gc.GarbageCollector(store=...).scan(...) / .collect(...). - The "Multiple Schemas" section is inverted: per-schema scoping makes any SUBSET of a store's schemas safe to collect — the old "always pass every schema or lose data" warning was the opposite of the shipped behavior and is replaced with a per-schema-attribution note. - Named-store examples bind the store at construction (no store_name= param). - Combined totals (orphaned/orphaned_bytes/referenced/stored) removed from the statistics reference and examples; readers sum the per-section fields. --- src/how-to/garbage-collection.md | 89 +++++++++++++++++--------------- src/how-to/manage-large-data.md | 9 ++-- 2 files changed, 52 insertions(+), 46 deletions(-) diff --git a/src/how-to/garbage-collection.md b/src/how-to/garbage-collection.md index c5ca416f..392ecffa 100644 --- a/src/how-to/garbage-collection.md +++ b/src/how-to/garbage-collection.md @@ -30,12 +30,16 @@ Run garbage collection periodically to reclaim storage space. ```python import datajoint as dj +# A collector is bound to one store (default store shown here) +collector = dj.gc.GarbageCollector() + # Scan for orphaned items (read-only) -stats = dj.gc.scan(schema1, schema2) -print(f"{stats['orphaned']} orphaned, {stats['orphaned_bytes'] / 1e6:.1f} MB reclaimable") +stats = collector.scan(schema1, schema2) +orphaned = stats['hash_orphaned'] + stats['schema_paths_orphaned'] +print(f"{orphaned} orphaned items") # Remove orphaned items -stats = dj.gc.collect(schema1, schema2, dry_run=False) +stats = collector.collect(schema1, schema2, dry_run=False) print(f"Deleted {stats['deleted']} items, freed {stats['bytes_freed'] / 1e6:.1f} MB") ``` @@ -45,11 +49,13 @@ Always scan first to see what would be deleted: ```python # Check what's orphaned -stats = dj.gc.scan(my_schema) +collector = dj.gc.GarbageCollector() +stats = collector.scan(my_schema) print(f"Hash-addressed orphaned: {stats['hash_orphaned']}") print(f"Schema paths orphaned: {stats['schema_paths_orphaned']}") -print(f"Total bytes: {stats['orphaned_bytes'] / 1e6:.1f} MB") +bytes_reclaimable = stats['hash_orphaned_bytes'] + stats['schema_paths_orphaned_bytes'] +print(f"Reclaimable: {bytes_reclaimable / 1e6:.1f} MB") ``` ## Dry Run Mode @@ -58,50 +64,50 @@ The default `dry_run=True` reports what would be deleted without deleting: ```python # Safe: reports how many items would be deleted (deletes nothing). -# For the per-item paths and reclaimable bytes, use dj.gc.scan() instead — +# For the per-item paths and reclaimable bytes, use scan() instead — # collect(dry_run=True) reports bytes_freed as 0 because nothing was removed. -stats = dj.gc.collect(my_schema, dry_run=True) -print(f"{stats['orphaned']} items would be deleted") +collector = dj.gc.GarbageCollector() +stats = collector.collect(my_schema, dry_run=True) +print(f"{stats['hash_orphaned'] + stats['schema_paths_orphaned']} items would be deleted") # After review, actually delete -stats = dj.gc.collect(my_schema, dry_run=False) +stats = collector.collect(my_schema, dry_run=False) ``` ## Multiple Schemas -If your data spans multiple schemas, scan all of them together: +Every managed path embeds the schema name (`{hash_prefix}/{schema}/...` and +`{schema_prefix}/{schema}/...`), so garbage collection is **per-schema**: each +schema is scanned against its own subtree. You may pass any subset of the +schemas sharing a store — a schema not passed is simply not scanned; its live +objects are never seen and never at risk. ```python -# Important: include ALL schemas that might share storage -stats = dj.gc.collect( - schema_raw, - schema_processed, - schema_analysis, - dry_run=False -) +collector = dj.gc.GarbageCollector() + +# Clean several schemas at once... +stats = collector.collect(schema_raw, schema_processed, schema_analysis, dry_run=False) + +# ...or just one — safe even when others share the same store +stats = collector.collect(schema_raw, dry_run=False) ``` -!!! warning "Per-schema paths and multi-schema stores" - Every hash-addressed path embeds the schema name - (`{hash_prefix}/{schema}/{hash}`), so deduplication is scoped **within** - each schema and every stored object is attributable to the schema that - wrote it. Orphan detection, however, lists the **entire store** on the - storage side: a schema that uses the store but is not passed to - `scan()`/`collect()` has its live objects misclassified as orphans — - and deleted. Always pass **every** schema that uses the store. The same - per-schema attribution is what makes leftovers of fully dropped schemas - identifiable and reclaimable by a later `collect()`. +!!! note "Per-schema attribution" + Because every path embeds its schema, deduplication is scoped within each + schema and every stored object is attributable to the schema that wrote it. + That is also what lets a later `collect()` reclaim the leftovers of a fully + dropped schema — pass a schema object bound to that (now empty) database. ## Named Stores If you use multiple named stores, specify which to clean: ```python -# Clean specific store -stats = dj.gc.collect(my_schema, store_name='archive', dry_run=False) +# Clean a specific store — bind the collector to it +stats = dj.gc.GarbageCollector(store='archive').collect(my_schema, dry_run=False) -# Or clean default store -stats = dj.gc.collect(my_schema, dry_run=False) # uses default store +# Or the default store +stats = dj.gc.GarbageCollector().collect(my_schema, dry_run=False) ``` ## Verbose Mode @@ -109,17 +115,17 @@ stats = dj.gc.collect(my_schema, dry_run=False) # uses default store See detailed progress: ```python -stats = dj.gc.collect( +stats = dj.gc.GarbageCollector().collect( my_schema, dry_run=False, - verbose=True # logs each deletion + verbose=True, # logs each deletion ) ``` ## Understanding the Statistics ```python -stats = dj.gc.scan(my_schema) +stats = dj.gc.GarbageCollector().scan(my_schema) # Hash-addressed storage (, , ) stats['hash_referenced'] # Items still in database @@ -133,11 +139,8 @@ stats['schema_paths_stored'] # Paths in storage stats['schema_paths_orphaned'] # Unreferenced paths stats['schema_paths_orphaned_bytes'] -# Totals -stats['referenced'] # Total referenced items -stats['stored'] # Total stored items -stats['orphaned'] # Total orphaned items -stats['orphaned_bytes'] +# Combine the two sections yourself if you want a grand total, e.g. +# stats['hash_orphaned'] + stats['schema_paths_orphaned'] ``` ## Scheduled Collection @@ -149,10 +152,10 @@ Run GC periodically in production: import datajoint as dj from myproject import schema1, schema2, schema3 -stats = dj.gc.collect( +stats = dj.gc.GarbageCollector().collect( schema1, schema2, schema3, dry_run=False, - verbose=True + verbose=True, ) if stats['errors'] > 0: @@ -216,10 +219,10 @@ DataJoint uses two storage patterns: ```python # Wrong -dj.gc.scan() +dj.gc.GarbageCollector().scan() # Right -dj.gc.scan(my_schema) +dj.gc.GarbageCollector().scan(my_schema) ``` ### Storage not decreasing diff --git a/src/how-to/manage-large-data.md b/src/how-to/manage-large-data.md index 555f5778..902451c9 100644 --- a/src/how-to/manage-large-data.md +++ b/src/how-to/manage-large-data.md @@ -115,12 +115,15 @@ import datajoint as dj # Objects are NOT automatically deleted with rows (MyTable & old_data).delete() +# A collector is bound to one store +collector = dj.gc.GarbageCollector() + # Scan for orphaned items (read-only) -stats = dj.gc.scan(my_schema) -print(f"{stats['orphaned']} orphaned, {stats['orphaned_bytes'] / 1e6:.1f} MB reclaimable") +stats = collector.scan(my_schema) +print(f"{stats['hash_orphaned'] + stats['schema_paths_orphaned']} orphaned items") # Remove orphaned items -stats = dj.gc.collect(my_schema, dry_run=False) +stats = collector.collect(my_schema, dry_run=False) ``` See [Clean Up Object Storage](garbage-collection.md) for details. From bdac4ed7da4b5f389f94b844e46bd318a6d738f5 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 6 Jul 2026 17:21:35 -0500 Subject: [PATCH 4/6] docs(gc): schemas in the GarbageCollector constructor; collect(dry_run) replaces scan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follows datajoint/datajoint-python#1479: the collector takes its schemas at construction (dj.gc.GarbageCollector(schema1, schema2, store=...)), and the separate scan() is gone — collect(dry_run=True) is the default read-only report (full orphan paths + reclaimable bytes), collect(dry_run=False) deletes. --- src/how-to/garbage-collection.md | 56 +++++++++++++++----------------- src/how-to/manage-large-data.md | 10 +++--- 2 files changed, 31 insertions(+), 35 deletions(-) diff --git a/src/how-to/garbage-collection.md b/src/how-to/garbage-collection.md index 392ecffa..0dde8772 100644 --- a/src/how-to/garbage-collection.md +++ b/src/how-to/garbage-collection.md @@ -30,16 +30,16 @@ Run garbage collection periodically to reclaim storage space. ```python import datajoint as dj -# A collector is bound to one store (default store shown here) -collector = dj.gc.GarbageCollector() +# A collector is bound to its schemas and one store (default store shown here) +collector = dj.gc.GarbageCollector(schema1, schema2) -# Scan for orphaned items (read-only) -stats = collector.scan(schema1, schema2) +# Read-only report — collect(dry_run=True) is the default and deletes nothing +stats = collector.collect() orphaned = stats['hash_orphaned'] + stats['schema_paths_orphaned'] print(f"{orphaned} orphaned items") -# Remove orphaned items -stats = collector.collect(schema1, schema2, dry_run=False) +# Actually remove the orphaned items +stats = collector.collect(dry_run=False) print(f"Deleted {stats['deleted']} items, freed {stats['bytes_freed'] / 1e6:.1f} MB") ``` @@ -48,9 +48,9 @@ print(f"Deleted {stats['deleted']} items, freed {stats['bytes_freed'] / 1e6:.1f} Always scan first to see what would be deleted: ```python -# Check what's orphaned -collector = dj.gc.GarbageCollector() -stats = collector.scan(my_schema) +# Preview: the default dry_run=True reports without deleting +collector = dj.gc.GarbageCollector(my_schema) +stats = collector.collect() print(f"Hash-addressed orphaned: {stats['hash_orphaned']}") print(f"Schema paths orphaned: {stats['schema_paths_orphaned']}") @@ -63,15 +63,15 @@ print(f"Reclaimable: {bytes_reclaimable / 1e6:.1f} MB") The default `dry_run=True` reports what would be deleted without deleting: ```python -# Safe: reports how many items would be deleted (deletes nothing). -# For the per-item paths and reclaimable bytes, use scan() instead — -# collect(dry_run=True) reports bytes_freed as 0 because nothing was removed. -collector = dj.gc.GarbageCollector() -stats = collector.collect(my_schema, dry_run=True) +# dry_run=True (the default) is a full read-only report — it lists the exact +# orphaned paths and reclaimable bytes and deletes nothing. Only bytes_freed +# and the deleted counts stay 0 until you actually collect. +collector = dj.gc.GarbageCollector(my_schema) +stats = collector.collect() # dry_run=True print(f"{stats['hash_orphaned'] + stats['schema_paths_orphaned']} items would be deleted") # After review, actually delete -stats = collector.collect(my_schema, dry_run=False) +stats = collector.collect(dry_run=False) ``` ## Multiple Schemas @@ -83,13 +83,11 @@ schemas sharing a store — a schema not passed is simply not scanned; its live objects are never seen and never at risk. ```python -collector = dj.gc.GarbageCollector() - # Clean several schemas at once... -stats = collector.collect(schema_raw, schema_processed, schema_analysis, dry_run=False) +stats = dj.gc.GarbageCollector(schema_raw, schema_processed, schema_analysis).collect(dry_run=False) # ...or just one — safe even when others share the same store -stats = collector.collect(schema_raw, dry_run=False) +stats = dj.gc.GarbageCollector(schema_raw).collect(dry_run=False) ``` !!! note "Per-schema attribution" @@ -104,10 +102,10 @@ If you use multiple named stores, specify which to clean: ```python # Clean a specific store — bind the collector to it -stats = dj.gc.GarbageCollector(store='archive').collect(my_schema, dry_run=False) +stats = dj.gc.GarbageCollector(my_schema, store='archive').collect(dry_run=False) # Or the default store -stats = dj.gc.GarbageCollector().collect(my_schema, dry_run=False) +stats = dj.gc.GarbageCollector(my_schema).collect(dry_run=False) ``` ## Verbose Mode @@ -115,8 +113,7 @@ stats = dj.gc.GarbageCollector().collect(my_schema, dry_run=False) See detailed progress: ```python -stats = dj.gc.GarbageCollector().collect( - my_schema, +stats = dj.gc.GarbageCollector(my_schema).collect( dry_run=False, verbose=True, # logs each deletion ) @@ -125,7 +122,7 @@ stats = dj.gc.GarbageCollector().collect( ## Understanding the Statistics ```python -stats = dj.gc.GarbageCollector().scan(my_schema) +stats = dj.gc.GarbageCollector(my_schema).collect() # dry_run=True default # Hash-addressed storage (, , ) stats['hash_referenced'] # Items still in database @@ -152,8 +149,7 @@ Run GC periodically in production: import datajoint as dj from myproject import schema1, schema2, schema3 -stats = dj.gc.GarbageCollector().collect( - schema1, schema2, schema3, +stats = dj.gc.GarbageCollector(schema1, schema2, schema3).collect( dry_run=False, verbose=True, ) @@ -218,11 +214,11 @@ DataJoint uses two storage patterns: ### "At least one schema must be provided" ```python -# Wrong -dj.gc.GarbageCollector().scan() +# Wrong — no schemas +dj.gc.GarbageCollector() -# Right -dj.gc.GarbageCollector().scan(my_schema) +# Right — schemas go in the constructor +dj.gc.GarbageCollector(my_schema).collect() ``` ### Storage not decreasing diff --git a/src/how-to/manage-large-data.md b/src/how-to/manage-large-data.md index 902451c9..4fb93089 100644 --- a/src/how-to/manage-large-data.md +++ b/src/how-to/manage-large-data.md @@ -115,15 +115,15 @@ import datajoint as dj # Objects are NOT automatically deleted with rows (MyTable & old_data).delete() -# A collector is bound to one store -collector = dj.gc.GarbageCollector() +# A collector is bound to its schemas and one store +collector = dj.gc.GarbageCollector(my_schema) -# Scan for orphaned items (read-only) -stats = collector.scan(my_schema) +# Read-only report (collect defaults to dry_run=True) +stats = collector.collect() print(f"{stats['hash_orphaned'] + stats['schema_paths_orphaned']} orphaned items") # Remove orphaned items -stats = collector.collect(my_schema, dry_run=False) +stats = collector.collect(dry_run=False) ``` See [Clean Up Object Storage](garbage-collection.md) for details. From 0467e6aea0cccf5506f319f3f985b0955127c660 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 6 Jul 2026 17:32:36 -0500 Subject: [PATCH 5/6] docs(gc): consistent hash-side stat names (hash_paths_*, orphaned_hash_paths) Mirror the schema-addressed naming on the hash side of the collect() report (datajoint/datajoint-python#1479): hash_referenced->hash_paths_referenced, hash_stored->hash_paths_stored, hash_orphaned->hash_paths_orphaned, hash_orphaned_bytes->hash_paths_orphaned_bytes, orphaned_hashes->orphaned_hash_paths. --- src/how-to/garbage-collection.md | 18 +++++++++--------- src/how-to/manage-large-data.md | 2 +- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/src/how-to/garbage-collection.md b/src/how-to/garbage-collection.md index 0dde8772..11c4f87a 100644 --- a/src/how-to/garbage-collection.md +++ b/src/how-to/garbage-collection.md @@ -35,7 +35,7 @@ collector = dj.gc.GarbageCollector(schema1, schema2) # Read-only report — collect(dry_run=True) is the default and deletes nothing stats = collector.collect() -orphaned = stats['hash_orphaned'] + stats['schema_paths_orphaned'] +orphaned = stats['hash_paths_orphaned'] + stats['schema_paths_orphaned'] print(f"{orphaned} orphaned items") # Actually remove the orphaned items @@ -52,9 +52,9 @@ Always scan first to see what would be deleted: collector = dj.gc.GarbageCollector(my_schema) stats = collector.collect() -print(f"Hash-addressed orphaned: {stats['hash_orphaned']}") +print(f"Hash-addressed orphaned: {stats['hash_paths_orphaned']}") print(f"Schema paths orphaned: {stats['schema_paths_orphaned']}") -bytes_reclaimable = stats['hash_orphaned_bytes'] + stats['schema_paths_orphaned_bytes'] +bytes_reclaimable = stats['hash_paths_orphaned_bytes'] + stats['schema_paths_orphaned_bytes'] print(f"Reclaimable: {bytes_reclaimable / 1e6:.1f} MB") ``` @@ -68,7 +68,7 @@ The default `dry_run=True` reports what would be deleted without deleting: # and the deleted counts stay 0 until you actually collect. collector = dj.gc.GarbageCollector(my_schema) stats = collector.collect() # dry_run=True -print(f"{stats['hash_orphaned'] + stats['schema_paths_orphaned']} items would be deleted") +print(f"{stats['hash_paths_orphaned'] + stats['schema_paths_orphaned']} items would be deleted") # After review, actually delete stats = collector.collect(dry_run=False) @@ -125,10 +125,10 @@ stats = dj.gc.GarbageCollector(my_schema).collect( stats = dj.gc.GarbageCollector(my_schema).collect() # dry_run=True default # Hash-addressed storage (, , ) -stats['hash_referenced'] # Items still in database -stats['hash_stored'] # Items in storage -stats['hash_orphaned'] # Unreferenced (can be deleted) -stats['hash_orphaned_bytes'] # Size of orphaned items +stats['hash_paths_referenced'] # Items still in database +stats['hash_paths_stored'] # Items in storage +stats['hash_paths_orphaned'] # Unreferenced (can be deleted) +stats['hash_paths_orphaned_bytes'] # Size of orphaned items # Schema-addressed storage (, ) stats['schema_paths_referenced'] # Paths still in database @@ -137,7 +137,7 @@ stats['schema_paths_orphaned'] # Unreferenced paths stats['schema_paths_orphaned_bytes'] # Combine the two sections yourself if you want a grand total, e.g. -# stats['hash_orphaned'] + stats['schema_paths_orphaned'] +# stats['hash_paths_orphaned'] + stats['schema_paths_orphaned'] ``` ## Scheduled Collection diff --git a/src/how-to/manage-large-data.md b/src/how-to/manage-large-data.md index 4fb93089..19c5715c 100644 --- a/src/how-to/manage-large-data.md +++ b/src/how-to/manage-large-data.md @@ -120,7 +120,7 @@ collector = dj.gc.GarbageCollector(my_schema) # Read-only report (collect defaults to dry_run=True) stats = collector.collect() -print(f"{stats['hash_orphaned'] + stats['schema_paths_orphaned']} orphaned items") +print(f"{stats['hash_paths_orphaned'] + stats['schema_paths_orphaned']} orphaned items") # Remove orphaned items stats = collector.collect(dry_run=False) From cbd399086da2e09b8be0ce03d3dafb94782e5cde Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Tue, 7 Jul 2026 18:56:24 -0500 Subject: [PATCH 6/6] Reconcile residual per-schema and path-form inconsistencies - type-system.md : fix stale intro path to {hash_prefix}/{schema}/{hash} matching the tree diagram and codec table - type-system.md : correct the two bullets that still claimed per-project scope / cross-schema sharing to per-schema, consistent with the Deduplication bullet - npy-codec.md: update the example and JSON metadata paths to the {schema_prefix}/.../{attribute}_{token}.npy form used elsewhere --- src/reference/specs/npy-codec.md | 4 ++-- src/reference/specs/type-system.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/reference/specs/npy-codec.md b/src/reference/specs/npy-codec.md index 7882f984..cda087c8 100644 --- a/src/reference/specs/npy-codec.md +++ b/src/reference/specs/npy-codec.md @@ -192,7 +192,7 @@ storage that reflects your data model. {schema_prefix}/{schema}/{table}/{primary_key_values}/{attribute}_{token}.npy ``` -Example: `lab_ephys/recording/recording_id=1/waveform.npy` +Example: `_schema/lab_ephys/recording/recording_id=1/waveform_x8f2a9b1.npy` This schema-addressed layout means you can browse the object store and understand the organization because it mirrors your database schema. @@ -203,7 +203,7 @@ The database column stores: ```json { - "path": "lab_ephys/recording/recording_id=1/waveform.npy", + "path": "_schema/lab_ephys/recording/recording_id=1/waveform_x8f2a9b1.npy", "store": "main", "dtype": "float64", "shape": [1000, 32] diff --git a/src/reference/specs/type-system.md b/src/reference/specs/type-system.md index a379e1ae..bae8440e 100644 --- a/src/reference/specs/type-system.md +++ b/src/reference/specs/type-system.md @@ -291,11 +291,11 @@ class ObjectCodec(SchemaCodec): Hash-addressed storage with deduplication for individual, atomic objects: -- **Hash-addressed**: Path derived from content hash: `_hash/{hash[:2]}/{hash[2:4]}/{hash}` +- **Hash-addressed**: Path derived from content hash, with the schema name embedded: `{hash_prefix}/{schema}/{hash}` - **Individual/atomic objects only**: Stores single files or serialized blobs (not directory structures) - Cannot handle complex multi-part objects like Zarr arrays—use `` for those -- **Per-project scope**: content is shared across all schemas in a project (not per-schema) -- Many-to-one: multiple rows (even across schemas) can reference same content +- **Per-schema scope**: content is deduplicated within each schema (the schema name is part of every hash path) +- Many-to-one: multiple rows within the same schema can reference the same content - Reference counted for garbage collection - Deduplication: identical content stored once **per schema** — the schema name is part of every hash path, which scopes deduplication and lets garbage collection attribute each stored object to its schema - **dtype**: `json` (stores hash, store name, size, metadata)