Skip to content

Commit 005fdd1

Browse files
committed
docs(codecs): fix stale storage-path and hash-algorithm docstrings
Three built-in codec class docstrings described pre-2.3.1 storage layouts (and HashCodec claimed the wrong hash algorithm entirely): - HashCodec: said 'SHA256'; actually MD5 + Base32 (hash_registry.py:65-67). Layout said '_hash/{hash[:2]}/{hash[2:4]}/{hash}'; actually '{hash_prefix}/{schema}/{hash}' with hash_prefix defaulting to '_hash' and no [:2]/[:4] split anywhere in code. - NpyCodec: layout said '{schema}/{table}/{pk}/{attribute}.npy'; actually '{schema_prefix}/{schema}/{table}/{pk}/{attribute}_{token}.npy' after #1479 added the schema_prefix section and the random token suffix (via storage.build_object_path). - ObjectCodec: same drift as NpyCodec, as a directory rather than a file. All three docstrings had their 'Deletion:' line updated in v2.3.1 (dj.gc.collect() -> dj.gc.GarbageCollector) but the surrounding path claims were left stale. Docs-only.
1 parent af76ab9 commit 005fdd1

3 files changed

Lines changed: 14 additions & 8 deletions

File tree

src/datajoint/builtin_codecs/hash.py

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
"""
2-
Hash-addressed storage codec with SHA256 deduplication.
2+
Hash-addressed storage codec with MD5-based deduplication.
33
"""
44

55
from __future__ import annotations
@@ -12,11 +12,13 @@
1212

1313
class HashCodec(Codec):
1414
"""
15-
Hash-addressed storage with SHA256 deduplication.
15+
Hash-addressed storage with content-addressed deduplication.
1616
1717
The ``<hash@>`` codec stores raw bytes using hash-addressed storage.
18-
Data is identified by its SHA256 hash and stored in a hierarchical directory:
19-
``_hash/{hash[:2]}/{hash[2:4]}/{hash}``
18+
Data is identified by a 26-character Base32-encoded MD5 digest of the
19+
content, and stored at ``{hash_prefix}/{schema}/{hash}`` (``hash_prefix``
20+
defaults to ``_hash``). Stores with subfolding configured insert
21+
additional path segments: ``{hash_prefix}/{schema}/{fold1}/{fold2}/{hash}``.
2022
2123
The database column stores JSON metadata: ``{hash, store, size}``.
2224
Duplicate content is automatically deduplicated across all tables.

src/datajoint/builtin_codecs/npy.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -226,7 +226,9 @@ class NpyCodec(SchemaCodec):
226226
Schema-addressed storage for numpy arrays as .npy files.
227227
228228
The ``<npy@>`` codec stores numpy arrays as standard ``.npy`` files
229-
using schema-addressed paths: ``{schema}/{table}/{pk}/{attribute}.npy``.
229+
using schema-addressed paths: ``{schema_prefix}/{schema}/{table}/{pk}/{attribute}_{token}.npy``
230+
(``schema_prefix`` defaults to ``_schema``; ``{token}`` is a random
231+
per-write suffix).
230232
Arrays are fetched lazily via ``NpyRef``, which provides metadata access
231233
without I/O and transparent numpy integration via ``__array__``.
232234
@@ -269,7 +271,7 @@ class Recording(dj.Manual):
269271
270272
Storage Details:
271273
- File format: NumPy .npy (version 1.0 or 2.0)
272-
- Path: ``{schema}/{table}/{pk}/{attribute}.npy``
274+
- Path: ``{schema_prefix}/{schema}/{table}/{pk}/{attribute}_{token}.npy``
273275
- Database column: JSON with ``{path, store, dtype, shape}``
274276
275277
Deletion: Requires garbage collection via ``dj.gc.GarbageCollector``.

src/datajoint/builtin_codecs/object.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,9 @@ class ObjectCodec(SchemaCodec):
1515
Schema-addressed storage for files and folders.
1616
1717
The ``<object@>`` codec provides managed file/folder storage using
18-
schema-addressed paths: ``{schema}/{table}/{pk}/{field}/``. This creates
18+
schema-addressed paths: ``{schema_prefix}/{schema}/{table}/{pk}/{field}_{token}/``
19+
(``schema_prefix`` defaults to ``_schema``; ``{token}`` is a random
20+
per-write suffix). This creates
1921
a browsable organization in object storage that mirrors the database schema.
2022
2123
Unlike hash-addressed storage (``<hash@>``), each row has its own unique path
@@ -51,7 +53,7 @@ def make(self, key):
5153
Storage Structure:
5254
Objects are stored at::
5355
54-
{store_root}/{schema}/{table}/{pk}/{field}/
56+
{store_root}/{schema_prefix}/{schema}/{table}/{pk}/{field}_{token}/
5557
5658
Deletion: Requires garbage collection via ``dj.gc.GarbageCollector``.
5759

0 commit comments

Comments
 (0)