DataLad + git-annex workflow

datalad.org · Storage · 1 record · JSON

DataLad + git-annex workflow (orchestration layer) Storage

caveatinstitution?card?R

handbook.datalad.org/en/latest/basics/101-139-s3.html · academic · storage/datalad-git-annex-workflow · JSON · compare

Free tier	N/A — DataLad is a wrapper over git-annex; quota is the special remote's. Its value here is that it makes 'dataset + N special remotes + reproducible publish' a supported, documented workflow instead of hand-written annex commands. unit mixed; metered on unstated
Price	none stated
Per-object cap	not stated
Retention	not stated
Egress	Inherited; DataLad's S3 walkthrough specifically configures `publicurl` so consumers clone the git metadata from GitHub and stream file content from the bucket with no credentials.
Identity	institution (placeholder) inferred from the category 'academic'; the source record states no identity cost, so this is a class assumption, not evidence
Binding	none no constraint in the record binds at this scale; the record carries no limit of a kind the vocabulary names
Interface	s3, git, cli s3
Tooling	`datalad` CLI (Python), `datalad create-sibling-s3`, `datalad push --to <sibling>`, `datalad get`. Requires git-annex >= 10.20230802 for S3 publicurl on post-April-2023 buckets.
Composes with	DataLad's 'dataset hierarchy' (superdataset + subdatasets) maps a 20 TiB archive into hundreds of independently-served subdatasets, each with its own sibling/remote. Combined with `datalad siblings` this is the cleanest way to shard one logical corpus across many free S3 accounts while keeping one coherent git history.
Reliability	Used in production by academic consortia (OpenNeuro etc.) precisely for multi-TB datasets on donated storage — the most battle-tested 'big dataset on free/cheap stores' pattern that exists.
Notes	N/A — DataLad is a wrapper over git-annex; quota is the special remote's. Its value here is that it makes 'dataset + N special remotes + reproducible publish' a supported, documented workflow instead of hand-written annex commands.
Evidence rung	reported One page read moves it to documented.
Checked	2026-10-21 (inherited); stale on 2026-11-04

Evidence (2)

date	source	what it proved
2026-10-21	handbook.datalad.org/en/latest/basics/101-139-s3.html	'This walk-through requires git-annex >= 10.20230802 ... Prior versions do not support public access via the publicurl parameter with S3 buckets created after April 2023'
2026-10-21	handbook.datalad.org/en/latest/basics/101-139-gitlfs.html	DataLad's Git LFS special-remote walkthrough on GitHub (linked as next chapter)

Unknown or withheld values (6)

$.free_tier.notes process
$.identity_cost.kind identity-defaulted
$.per_object_cap.notes not-extracted
$.per_object_cap.stated not-extracted
$.retention.notes not-extracted
$.retention.stated not-extracted

Built from commit 872967ae41, data as of 2026-10-27. Every value links to its source; freshness badges are computed in your browser. How to read this.