fshy

fshy / Storage

Mozilla Data Collective

mozilladatacollective.com · Storage · 1 record · JSON

Mozilla Data Collective (MDC) Storage

caveatemailno cardRper-object-cap

mozilladatacollective.com · research-data-repository · storage/mozilla-data-collective · JSON · compare

Free tier150 GB per uploaded file — raised from a lower value in the Python SDK on 2026-03-24 ('Increase upload file size limit to 150GB'). No published per-account or total quota anywhere in the provider terms, FAQ, or API reference; the archive in the public catalogue runs to multi-GB objects (one measured dataset is 9,786,006,314 bytes = 9.79 GB as a single .tar.gz). Uploads are multipart with a hard 10,000-part ceiling, so a 1 TB object would need ≥100 MB parts — the SDK enforces this and names the minimum part size rather than failing mid-upload.
unit GB; metered on unstated
Pricenone stated
Per-object cap150 GB per upload (150000000000 bytes)
Retentionnot stated
EgressFree to the consumer; storage is Cloudflare R2 in the EU plus AWS in Ireland per their own subprocessor table. Downloads are short-lived presigned URLs, and downloads are rate-limited to 30 per organisation per day (429 beyond that).
Identityemail email
Limits
  • request-rate 30 downloads/day source
  • per-object-cap 150 GB per upload source
  • pull-count 30 downloads/day source
  • write-gated review every request to upload' — you file a Request to Upload, wait days, and are int source
  • write-gated skipping the approval flow ends the account source
write-gated, hard. 'we manually review every request to upload' — you file a Request to Upload, wait days, and are interviewed by email about 'yourself, the organization you are representing, the data you want to upload and what are your goals'. Every dataset must carry a datasheet. API upload calls are rate-limited at their discretion and skipping the approval flow ends the account.
Bindingper-object-cap the per-object cap sets the shard size and therefore the object count; it bounds the total more tightly than any headline quota
Interfaces3, rest rest + s3
ToolingBest-in-slot for a scripted pipeline: a documented REST API at https://mozilladatacollective.com/api (bearer API key), plus a first-party Python SDK with resumable multipart uploads, configurable part_size and max_workers, and a .mdc-sample-upload.json resume-state file. Verified live: GET /api/health -> 200 (15 bytes), GET /api/datasets?limit=1 -> 200 with full metadata including sizeBytes and a SHA-256 checksum field.
Composes withOne dataset = one submission = one archive file plus one sample file. Nothing in the published terms caps the number of submissions per account, so the multiplier is submissions. The catalogue is browsable and enumerable anonymously through /api/datasets, which means the read side can be driven entirely from a script.
ReliabilityThe gate is a human reviewer, not a form — treat acceptance as the scarce resource. 30 downloads/day/org will throttle any mirroring plan. Hash field is present in metadata, so a checksum roundtrip is cheap to automate.
Notes150 GB per uploaded file — raised from a lower value in the Python SDK on 2026-03-24 ('Increase upload file size limit to 150GB'). No published per-account or total quota anywhere in the provider terms, FAQ, or API reference; the archive in the public catalogue runs to multi-GB objects (one measured dataset is 9,786,006,314 bytes = 9.79 GB as a single .tar.gz). Uploads are multipart with a hard 10,000-part ceiling, so a 1 TB object would need ≥100 MB parts — the SDK enforces this and names the minimum part size rather than failing mid-upload.
Evidence rungreported One page read moves it to documented.
Checked2026-10-21 (inherited); stale on 2026-11-04
Parsed numbers
  • $.limits[1].value limit:per-object-cap 150.0 GB
Evidence (7)
datesourcewhat it proved
2026-10-21community.mozilladatacollective.com/uploading-your-dataset-topened: 'In order to ensure that the datasets hosted in our platform are aligned with the Mozilla Data Collective values & manifesto we manually review every request to upload'; 'please allow for a few days to get back to you'
2026-10-21github.com/Mozilla-Data-Collective/datacollective-python/comopened: commit 'Increase upload file size limit to 150GB' dated 2026-03-24. This is the only published per-file number and it is a code change, not marketing copy.
2026-10-21mozilla-data-collective.github.io/datacollective-python/uploopened: 'A single upload can have at most 10,000 parts … a 1 TB file needs parts of at least ~100 MB'; multipart, resumable, part floor 5 MB.
2026-10-21mozilladatacollective.com/api-reference/docsopened: POST /datasets/{datasetId}/download returns a presigned downloadUrl with expiresAt, sizeBytes and checksum; 'Organizations are limited to 30 dataset downloads per day … 429'; 'Returns a presigned storage URL.'
2026-10-21mozilladatacollective.com/terms/providersopened: 'MDC reserves the right to implement and enforce limits on MDC API upload calls'; subprocessor table lists 'Cloudflare R2 — Dataset storage of uploaded files — EU' and AWS Ireland.
2026-10-21mozilladatacollective.com/api/healthPROBED: curl -> HTTP 200, 15 bytes. curl https://mozilladatacollective.com/api/datasets?limit=1 -> HTTP 200, 1,626 bytes, real JSON. The API is live and needs no key for the catalogue.
2026-10-21withheldPROBED: GET /api/datasets/bibletts-luo-aligned-bible-speech-4118962a -> 200, datasetUrl, filename bibletts-luo-aligned-bible-speech-4118962a.tar.gz, sizeBytes 9786006314, checksum 221a7218f72a8fd106459d95dfd9341d3ff626e7456c2912336c0047bc96143f, isPaid false. Proves multi-GB single-object hosting actually exists on the platform.
Unknown or withheld values (5)
  • $.evidence[6].url private-url
  • $.free_tier.notes process
  • $.per_object_cap.notes not-extracted
  • $.retention.notes not-extracted
  • $.retention.stated not-extracted