Skip to content
8 changes: 4 additions & 4 deletions src/explanation/type-system.md
Original file line number Diff line number Diff line change
Expand Up @@ -266,14 +266,14 @@ result = np.mean(ref) # Downloads automatically

### `<object@>` — Schema-Addressed Storage

Schema-addressed storage for files and folders. Path mirrors the database structure: `{schema}/{table}/{pk}/{attribute}`.
Schema-addressed storage for files and folders. Path mirrors the database structure within the store's schema-addressed section: `{schema_prefix}/{schema}/{table}/{pk}/{attribute}_{token}` (`schema_prefix` defaults to `_schema`).

```python
class RecordingAnalysis(dj.Computed):
definition = """
-> Recording
---
results : <object@> # Stored at {schema}/{table}/{pk}/results/
results : <object@> # Stored at _schema/{schema}/{table}/{pk}/results_{token}/
"""
```

Expand All @@ -296,9 +296,9 @@ class RawData(dj.Manual):

Object store codecs use one of two addressing schemes:

**Hash-addressed** — Path derived from content hash (e.g., `_hash/ab/cd/abcd1234...`). Provides automatic deduplication—identical content stored once. Used by `<blob@>`, `<attach@>`, `<hash@>`.
**Hash-addressed** — Path derived from content hash, with the schema name embedded (e.g., `_hash/{schema}/ab/cd/abcd1234...`; section configurable via `hash_prefix`). Provides automatic deduplication—identical content stored once **per schema**. Used by `<blob@>`, `<attach@>`, `<hash@>`.

**Schema-addressed** — Path mirrors database structure: `{schema}/{table}/{pk}/{attribute}`. Human-readable, browsable paths that reflect your data organization. No deduplication. Used by `<object@>`, `<npy@>`, and plugin codecs (`<zarr@>`, `<figpack@>`, `<photon@>`).
**Schema-addressed** — Path mirrors database structure: `_schema/{schema}/{table}/{pk}/{attribute}_{token}` (section configurable via `schema_prefix`). Human-readable, browsable paths that reflect your data organization. No deduplication. Used by `<object@>`, `<npy@>`, and plugin codecs (`<zarr@>`, `<figpack@>`, `<photon@>`).

| Mode | Database | Object Store | Deduplication | Use Case |
|------|----------|--------------|---------------|----------|
Expand Down
2 changes: 1 addition & 1 deletion src/how-to/choose-storage-type.md
Original file line number Diff line number Diff line change
Expand Up @@ -396,7 +396,7 @@ file : <attach@> # File attachments

### Schema-Addressed: `<npy@>` or `<object@>`

**Storage:** Object store at `{store}/_schema/{schema}/{table}/{key}/{field}.{token}.ext`
**Storage:** Object store at `{store}/_schema/{schema}/{table}/{key}/{field}_{token}.ext`

**Syntax:**
```python
Expand Down
11 changes: 9 additions & 2 deletions src/how-to/configure-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ _hash/{schema}/ab/cd/abcdefghijklmnopqrstuvwxyz
Schema-addressed storage (`<object@>`, `<npy@>`) does not use subfolding—it uses key-based paths:

```
{location}/_schema/{partition}/{schema}/{table}/{key}/{field_name}.{token}.{ext}
{location}/_schema/{partition}/{schema}/{table}/{key}/{field_name}_{token}{ext}
```

### Filesystem Recommendations
Expand Down Expand Up @@ -282,7 +282,7 @@ This unified approach enables:

## Customizing Storage Sections

Each store is divided into sections for different storage types. By default, DataJoint uses `_hash/` for hash-addressed storage and `_schema/` for schema-addressed storage. You can customize the path prefix for each section using the `*_prefix` configuration parameters to map DataJoint to existing storage layouts:
Each store is divided into sections for different storage types. By default, DataJoint uses `_hash/` for hash-addressed storage and `_schema/` for schema-addressed storage. You can customize the path prefix for each section using the `*_prefix` configuration parameters to map DataJoint to existing storage layouts. Writers, garbage collection, and `<filepath@>` validation all read the same per-store values, so a relocated section stays consistent end to end:

```json
{
Expand All @@ -304,6 +304,13 @@ Each store is divided into sections for different storage types. By default, Dat
- The `hash_prefix` and `schema_prefix` sections are reserved for DataJoint-managed storage
- The `filepath_prefix` is optional (`null` = unrestricted, or set a required prefix)

!!! warning "Changing prefixes on an existing store"
Set the prefixes when the store is first configured. Changing them on a
store that already holds data leaves existing objects readable (their
paths are recorded in each row's metadata), but garbage collection scans
only the currently configured sections, and hash deduplication will not
match content stored under an old prefix.

**Example with hierarchical layout:**

```json
Expand Down
146 changes: 84 additions & 62 deletions src/how-to/garbage-collection.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,106 +30,114 @@ Run garbage collection periodically to reclaim storage space.
```python
import datajoint as dj

# Scan for orphaned items (dry run)
stats = dj.gc.scan(schema1, schema2)
print(dj.gc.format_stats(stats))
# A collector is bound to its schemas and one store (default store shown here)
collector = dj.gc.GarbageCollector(schema1, schema2)

# Remove orphaned items
stats = dj.gc.collect(schema1, schema2, dry_run=False)
print(dj.gc.format_stats(stats))
# Read-only report — collect(dry_run=True) is the default and deletes nothing
stats = collector.collect()
orphaned = stats['hash_paths_orphaned'] + stats['schema_paths_orphaned']
print(f"{orphaned} orphaned items")

# Actually remove the orphaned items
stats = collector.collect(dry_run=False)
print(f"Deleted {stats['deleted']} items, freed {stats['bytes_freed'] / 1e6:.1f} MB")
```

## Scan Before Collecting

Always scan first to see what would be deleted:

```python
# Check what's orphaned
stats = dj.gc.scan(my_schema)
# Preview: the default dry_run=True reports without deleting
collector = dj.gc.GarbageCollector(my_schema)
stats = collector.collect()

print(f"Hash-addressed orphaned: {stats['hash_orphaned']}")
print(f"Hash-addressed orphaned: {stats['hash_paths_orphaned']}")
print(f"Schema paths orphaned: {stats['schema_paths_orphaned']}")
print(f"Total bytes: {stats['orphaned_bytes'] / 1e6:.1f} MB")
bytes_reclaimable = stats['hash_paths_orphaned_bytes'] + stats['schema_paths_orphaned_bytes']
print(f"Reclaimable: {bytes_reclaimable / 1e6:.1f} MB")
```

## Dry Run Mode

The default `dry_run=True` reports what would be deleted without deleting:

```python
# Safe: shows what would be deleted
stats = dj.gc.collect(my_schema, dry_run=True)
print(dj.gc.format_stats(stats))
# dry_run=True (the default) is a full read-only report — it lists the exact
# orphaned paths and reclaimable bytes and deletes nothing. Only bytes_freed
# and the deleted counts stay 0 until you actually collect.
collector = dj.gc.GarbageCollector(my_schema)
stats = collector.collect() # dry_run=True
print(f"{stats['hash_paths_orphaned'] + stats['schema_paths_orphaned']} items would be deleted")

# After review, actually delete
stats = dj.gc.collect(my_schema, dry_run=False)
stats = collector.collect(dry_run=False)
```

## Multiple Schemas

If your data spans multiple schemas, scan all of them together:
Every managed path embeds the schema name (`{hash_prefix}/{schema}/...` and
`{schema_prefix}/{schema}/...`), so garbage collection is **per-schema**: each
schema is scanned against its own subtree. You may pass any subset of the
schemas sharing a store — a schema not passed is simply not scanned; its live
objects are never seen and never at risk.

```python
# Important: include ALL schemas that might share storage
stats = dj.gc.collect(
schema_raw,
schema_processed,
schema_analysis,
dry_run=False
)
# Clean several schemas at once...
stats = dj.gc.GarbageCollector(schema_raw, schema_processed, schema_analysis).collect(dry_run=False)

# ...or just one — safe even when others share the same store
stats = dj.gc.GarbageCollector(schema_raw).collect(dry_run=False)
```

!!! note "Per-schema deduplication"
Hash-addressed storage is deduplicated **within** each schema. Different
schemas have independent storage, so you only need to scan schemas that
share the same database.
!!! note "Per-schema attribution"
Because every path embeds its schema, deduplication is scoped within each
schema and every stored object is attributable to the schema that wrote it.
That is also what lets a later `collect()` reclaim the leftovers of a fully
dropped schema — pass a schema object bound to that (now empty) database.

## Named Stores

If you use multiple named stores, specify which to clean:

```python
# Clean specific store
stats = dj.gc.collect(my_schema, store_name='archive', dry_run=False)
# Clean a specific store — bind the collector to it
stats = dj.gc.GarbageCollector(my_schema, store='archive').collect(dry_run=False)

# Or clean default store
stats = dj.gc.collect(my_schema, dry_run=False) # uses default store
# Or the default store
stats = dj.gc.GarbageCollector(my_schema).collect(dry_run=False)
```

## Verbose Mode

See detailed progress:

```python
stats = dj.gc.collect(
my_schema,
stats = dj.gc.GarbageCollector(my_schema).collect(
dry_run=False,
verbose=True # logs each deletion
verbose=True, # logs each deletion
)
```

## Understanding the Statistics

```python
stats = dj.gc.scan(my_schema)
stats = dj.gc.GarbageCollector(my_schema).collect() # dry_run=True default

# Hash-addressed storage (<blob@>, <attach@>, <hash@>)
stats['hash_referenced'] # Items still in database
stats['hash_stored'] # Items in storage
stats['hash_orphaned'] # Unreferenced (can be deleted)
stats['hash_orphaned_bytes'] # Size of orphaned items
stats['hash_paths_referenced'] # Items still in database
stats['hash_paths_stored'] # Items in storage
stats['hash_paths_orphaned'] # Unreferenced (can be deleted)
stats['hash_paths_orphaned_bytes'] # Size of orphaned items

# Schema-addressed storage (<object@>, <npy@>)
stats['schema_paths_referenced'] # Paths still in database
stats['schema_paths_stored'] # Paths in storage
stats['schema_paths_orphaned'] # Unreferenced paths
stats['schema_paths_orphaned_bytes']

# Totals
stats['referenced'] # Total referenced items
stats['stored'] # Total stored items
stats['orphaned'] # Total orphaned items
stats['orphaned_bytes']
# Combine the two sections yourself if you want a grand total, e.g.
# stats['hash_paths_orphaned'] + stats['schema_paths_orphaned']
```

## Scheduled Collection
Expand All @@ -141,10 +149,9 @@ Run GC periodically in production:
import datajoint as dj
from myproject import schema1, schema2, schema3

stats = dj.gc.collect(
schema1, schema2, schema3,
stats = dj.gc.GarbageCollector(schema1, schema2, schema3).collect(
dry_run=False,
verbose=True
verbose=True,
)

if stats['errors'] > 0:
Expand All @@ -160,43 +167,58 @@ DataJoint uses two storage patterns:
### Hash-Addressed (`<blob@>`, `<attach@>`, `<hash@>`)

```
_hash/
{hash_prefix}/ # store setting, default: _hash/
{schema}/
ab/
cd/
abcdefghij... # Content identified by Base32-encoded MD5 hash
```

- Duplicate content shares storage within each schema
- The section location comes from the store's `hash_prefix` setting
(default `_hash`) — the same value the writer uses, so scanner and writer
cannot drift
- Duplicate content shares storage within each schema — the schema name is
part of every path, which is what scopes deduplication and lets GC
attribute each stored object to a schema
- Paths are stored in metadata—safe from config changes
- Cannot delete until no rows reference the content
- GC compares stored paths against filesystem
- GC compares stored paths against the filesystem

### Schema-Addressed (`<object@>`, `<npy@>`)

```
myschema/
mytable/
primary_key_values/
attribute_name/
data.zarr/
data.npy
{schema_prefix}/ # store setting, default: _schema/
{schema}/
{table}/
{primary_key_values}/
{attribute}_{token}.npy # single-file object
{attribute}_{token}.zarr/ # folder object (many files)
{attribute}_{token}.zarr.manifest.json
```

- Each row has unique path based on schema structure
- Paths mirror database organization
- GC removes paths not referenced by any row
- The section location comes from the store's `schema_prefix` setting
(default `_schema`). DataJoint 2.3.0 and earlier wrote these objects at
root level (`{schema}/...`); GC lists both layouts, so legacy objects
remain reclaimable
- Every write gets a unique random token, so multiple versions of a row's
object can coexist; GC reclaims superseded tokens
- Orphan matching is coverage-based: a stored file is live if it **is** a
referenced path, lies **under** a referenced folder object, or is its
`.manifest.json` sidecar — live folder objects are never partially
collected
- GC never touches the store's declared `filepath_prefix` namespace
(user-managed `<filepath@>` files)

## Troubleshooting

### "At least one schema must be provided"

```python
# Wrong
dj.gc.scan()
# Wrong — no schemas
dj.gc.GarbageCollector()

# Right
dj.gc.scan(my_schema)
# Right — schemas go in the constructor
dj.gc.GarbageCollector(my_schema).collect()
```

### Storage not decreasing
Expand Down
11 changes: 7 additions & 4 deletions src/how-to/manage-large-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,12 +115,15 @@ import datajoint as dj
# Objects are NOT automatically deleted with rows
(MyTable & old_data).delete()

# Scan for orphaned items
stats = dj.gc.scan(my_schema)
print(dj.gc.format_stats(stats))
# A collector is bound to its schemas and one store
collector = dj.gc.GarbageCollector(my_schema)

# Read-only report (collect defaults to dry_run=True)
stats = collector.collect()
print(f"{stats['hash_paths_orphaned'] + stats['schema_paths_orphaned']} orphaned items")

# Remove orphaned items
stats = dj.gc.collect(my_schema, dry_run=False)
stats = collector.collect(dry_run=False)
```

See [Clean Up Object Storage](garbage-collection.md) for details.
Expand Down
2 changes: 1 addition & 1 deletion src/how-to/manage-pipeline-project.md
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,7 @@ A DataJoint project creates a structured storage pattern:

### Considerations

- Object paths include schema name: `{project}/{schema}/{table}/...`
- Object paths include the schema name: `{project}/{schema_prefix}/{schema}/{table}/...` — the schema is embedded in every managed path (both sections), which scopes hash deduplication per schema and lets garbage collection attribute stored objects to schemas
- Users need read access to fetch blobs from upstream schemas
- Content-addressed storage (`<blob@>`) shares objects across tables
- Garbage collection requires coordinated delete permissions
Expand Down
2 changes: 1 addition & 1 deletion src/how-to/use-object-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ Table.insert1({'id': 2, 'array': data}) # References same object

`<npy@>` and `<object@>` use **schema-addressed** storage:

- Objects stored at paths that mirror database schema: `{schema}/{table}/{pk}/{attribute}.npy`
- Objects stored under the store's schema-addressed section at paths that mirror the database schema: `_schema/{schema}/{table}/{pk}/{attribute}_{token}.npy` (section configurable via `schema_prefix`)
- Browsable organization in object storage
- One object per entity (no deduplication)
- Supports lazy loading with metadata access
Expand Down
6 changes: 3 additions & 3 deletions src/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ Prefixes must be mutually exclusive (no prefix can be a parent/child of another)
**How storage methods use stores:**

- **Hash-addressed** (`<blob@>`, `<attach@>`): `{location}/{hash_prefix}/{schema}/{hash}` with optional subfolding
- **Schema-addressed** (`<object@>`, `<npy@>`): `{location}/{schema_prefix}/{partition}/{schema}/{table}/{key}/{field}.{token}.{ext}` with optional partitioning
- **Schema-addressed** (`<object@>`, `<npy@>`): `{location}/{schema_prefix}/{partition}/{schema}/{table}/{key}/{field}_{token}{ext}` with optional partitioning
- **Filepath** (`<filepath@>`): `{location}/{filepath_prefix}/{user_path}` (user-managed, cannot use hash or schema prefixes)

All storage methods share the same stores and default store. DataJoint reserves the configured `hash_prefix` and `schema_prefix` sections for managed storage; `<filepath@>` references can use any other paths (unless `filepath_prefix` is configured to restrict them).
Expand All @@ -127,12 +127,12 @@ All storage methods share the same stores and default store. DataJoint reserves
Without partitioning:
```
{location}/_hash/{schema}/ab/cd/abcd1234... # hash-addressed with subfolding
{location}/_schema/{schema}/{table}/{key}/data.x8f2a9b1.zarr # schema-addressed, no partitioning
{location}/_schema/{schema}/{table}/{key}/data_x8f2a9b1.zarr # schema-addressed, no partitioning
```

With `partition_pattern: "subject_id/session_date"`:
```
{location}/_schema/subject_id=042/session_date=2024-01-15/{schema}/{table}/{remaining_key}/data.x8f2a9b1.zarr
{location}/_schema/subject_id=042/session_date=2024-01-15/{schema}/{table}/{remaining_key}/data_x8f2a9b1.zarr
```

If table lacks partition attributes, it follows normal path structure.
Expand Down
Loading
Loading