Hugging Face Storage Buckets via S3 gateway (s3.hf.co) — as an annex/DVC backend
- Free allowance
- Public repos/buckets: 'Best-effort' — no published numeric ceiling on a Free account (private is a hard 100 GB).
- Verdict
- caveatusable, with a named limit or an unprobed read path
- Cost
- free tier
- Identity
- institution?placeholder, not evidence; card unknown, phone unknown
- Binding limit
- per-object-capsingle file will exceed 500GB
- Evidence
- reported4 sources; a third party said so
- Freshness
- stale on 2026-11-05checked 2026-10-21 (inherited); stale on 2026-11-05
| Free tier | Public repos/buckets: 'Best-effort' — no published numeric ceiling on a Free account (private is a hard 100 GB). Storage plans table: Free user or org = Public 'Best-effort', Private 100 GB. HF is on the known list, but the S3 gateway is new and turns HF into a generic S3 endpoint, which is the part that matters for this slot. unit GB; metered on stored-bytes |
|---|---|
| Price | none stated |
| Limits |
Bucket = not Git, so no git LFS semantics — you get S3 PUT/GET only. Hard limit: 'no single file will exceed 500GB' for Hub repos (buckets: gateway multipart, docs recommend 2 GB multipart chunks). Recommended <100k files and <10k entries per folder; >200 GB files discouraged due to CloudFront not caching them. Region pinned us-east-1. Gateway does not parse aws-chunked trailing checksums — must set request_checksum_calculation=when_required. |
| Binding | per-object-cap the per-object cap sets the shard size and therefore the object count; it bounds the total more tightly than any headline quota |
| Identity | institution (placeholder) inferred from the category 'academic'; the source record states no identity cost, so this is a class assumption, not evidence |
| Egress | Free and CDN-backed: 'Egress and CDN included at no extra cost'. Served via CloudFront, so downloads are cacheable. This is the single best egress term of any LFS-adjacent host found. |
| Per-object cap | single file will exceed 500GB (500000000000 bytes) |
| Retention | not stated |
| Interface | s3, git, cli part of the text of Free tier, above |
| Tooling | Any S3 client: AWS CLI, boto3, s5cmd; hf CLI (hf sync, hf buckets); python huggingface_hub. For this slot: git annex initremote type=S3 host=s3.hf.co region=us-east-1 requeststyle=path (annex needs path-style, which the gateway requires anyway) and dvc remote add ... ; dvc remote modify myremote endpointurl https://s3.hf.co/<namespace>. S3 creds are derived from an HF token via 'Generate S3 credentials' (HFAK... key). |
| Composes with | Bucket-per-purpose, unlimited buckets per namespace, unlimited free orgs. Because buckets are NON-versioned, git-annex supplies the versioning (hash-keyed objects) and DVC supplies the pointer layer — that combination gives you a versioned 500GB+ dataset store on a free, CDN-backed S3 endpoint. Dedupe is chunk-level on the Xet backend, so re-pushing near-identical files is cheap. |
| Reliability | Free public storage is explicitly 'best-effort' and HF says 'Beyond the first few gigabytes, please use this resource responsibly', with a mitigation the page describes as 'in place to prevent abuse', meaning a limit on excess use. That is the softest possible enforcement language, but also the least contractual. Gateway is single-region (us-east-1). |
| Parsed numbers |
|
Evidence 4
| date | source | what it proved |
|---|---|---|
| 2026-10-21 | huggingface.co/docs/hub/storage-buckets-s3 | endpoint https://s3.hf.co/<namespace>, region us-east-1, path addressing, HFAK credentials from an HF token |
| 2026-10-21 | huggingface.co/docs/hub/storage-limits | Free user or org: Public 'Best-effort', Private 100GB; 'no single file will exceed 500GB' |
| 2026-10-21 | huggingface.co/storage | 'Egress and CDN included at no extra cost' |
| 2026-10-21 | s3.hf.co/ | probe: GET -> HTTP 403 with x-amz-request-id (live S3-compatible gateway answering) |
Unknown values 3
$.per_object_cap.notesthe field may be stated in a source that was not read$.retention.notesthe field may be stated in a source that was not read$.retention.statedthe field may be stated in a source that was not read