Wikimedia Downloads

wikimedia.org · Debrid, Storage · 2 records · JSON

Wikimedia Downloads (dumps.wikimedia.org) Debridcaveatanonymousno cardMper-ip-shaping
Wikimedia Commons Storagecaveatemailno cardRper-object-cap

Wikimedia Downloads (dumps.wikimedia.org) Debrid

caveatanonymousno cardMper-ip-shaping

dumps.wikimedia.org/ · encyclopaedia corpus (monthly XML dumps, HTTP mirrors) · debrid/wikimedia-dumps · JSON · compare

Free tier	Complete public wiki dumps, monthly, free and anonymous. 'Downloads are also rate limited and capped at 3 connections per-IP.' The English Wikipedia article dump measured here is 25,800,511,704 bytes. unit unlimited; metered on none; The dumps are being reorganised: the XML dumps are 'deprecated and will be sunset in January 2027' in favour of MediaWiki Content File Exports.
Price	USD 0
Per-object cap	not applicable: one compressed file per dump. enwiki-latest-pages-articles.xml.bz2 is 25,800,511,704 bytes; ranged reads are supported (accept-ranges: bytes).
Retention	monthly snapshots are kept; 'latest' is replaced each run and older dated directories persist for a period the dump index does not quantify
Egress	Free, but conditioned on the User-Agent policy: 'If you are reading this on Wikimedia servers, please note that we are now enforcing the Wikimedia Foundation User-Agent Policy for dumps downloads. Downloads are also rate limited and capped at 3 connections per-IP ... Clients that do not respect the user-agent policy or who try to evade these limits may be blocked.' free_allowance: no byte allowance published
Identity	anonymous No account and no key; a descriptive User-Agent is the only requirement, and it is not a credential.
Limits	per-ip-shaping capped at 3 connections per IP, and rate limited; measured here at 1.661-2.617 MiB/s over one connection (three runs) and 2.647-7.649 MiB/s total over eight connections (three runs), so parallelism helped up to 3.6x and the cap did not refuse the eighth connection source The User-Agent policy is enforced: a client without a descriptive agent may be blocked, and evasion attempts are named as a blocking reason on the front page.
Binding	per-ip-shaping The documented 3-connection cap is a connection count, not a byte rate: eight parallel range reads all returned 206 and the best of them reached 7.649 MiB/s, which is about 645 GiB/day.
Interface	rest, rsync, cli HTTPS per file with Range support; rsync mirrors are listed on meta.wikimedia.org; the dump index at /backup-index.html gives every file with size and checksum. endpoint: https://dumps.wikimedia.org/enwiki/latest/
Tooling	curl/wget with a descriptive User-Agent, rsync against a mirror, and the checksum files that ship beside each dump.
Composes with	A fixed encyclopaedia corpus to join against Common Crawl's web text; the 24 GiB article dump is small enough to hold beside a dataset snapshot, and the mirror list is the escape hatch when the per-IP cap binds.
Reliability	The dump format is changing: XML dumps sunset in January 2027, so a pipeline built on pages-articles.xml has a dated migration to the Content File Exports format.
Notes	Caveat is the 2027 format change and the 3-connection rule. A single connection gives about 178 GiB/day at the mean, and the best eight-connection run reached 645 GiB/day - which the published cap says a client should not do. The whole English Wikipedia article dump is one day's single-connection allowance.
Evidence rung	measured The file size came from a HEAD and the rate from a 64 MiB ranged read, three runs at each concurrency. All eight parallel requests returned 206, so the documented 3-connection cap did not refuse them at the transport level - whether it is enforced by rate, by UA or only at peak is not established.
Checked	2026-10-24 (live); stale on 2026-11-23
Parsed numbers	`$.limits[0].value` limit:per-ip-shaping 2.617 MiB `$.price_per_tb_month` price-per-tb USD0.0 TB / month

Evidence (4)

date	source	what it proved
2026-10-23	dumps.wikimedia.org/	'Downloads are also rate limited and capped at 3 connections per-IP', User-Agent policy enforcement, and the January 2027 XML sunset notice
2026-10-23	probe:https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2 --head	200, content-length 25,800,511,704, accept-ranges: bytes, last-modified 2026-10-08
2026-10-23	probe:https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2 --range 0-67108863	64 MiB ranges at one connection: 2.617, 2.084 and 1.661 MiB/s across three runs, 206 with the range honoured; sha256 of the first range 3a74bf06ec319f7d702771523ec099593e7569f00b5bddfdbe3e76ddbc914ea7
2026-10-23	probe:https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2 --concurrency 8 --shard-bytes 67108864	eight parallel 206s totalling 2.647, 4.748 and 7.649 MiB/s against same-run one-connection baselines of 1.317, 2.918 and 2.337 MiB/s - parallelism helped, and no request was refused

More fields (6)

{
 "enwiki_articles_xml_bytes": 25800511704,
 "one_conn_mib_s": [
  2.617,
  2.084,
  1.661
 ],
 "eight_conn_mib_s": [
  2.647,
  4.748,
  7.649
 ],
 "connection_cap": 3,
 "range_sha256": "3a74bf06ec319f7d702771523ec099593e7569f00b5bddfdbe3e76ddbc914ea7",
 "xml_sunset": "2027-01"
}

Wikimedia Commons Storage

caveatemailno cardRper-object-cap

commons.wikimedia.org · open media repository (institutional-grade, Wikimedia Foundation) · storage/wikimedia-commons · JSON · compare

Free tier	NO TOTAL QUOTA — read verbatim from Commons:Maximum_file_size: 'there is no limit to the total size of uploads by an individual account.' Per-file cap is 5 GiB (5,368,709,120 bytes) via chunked upload, 100 MiB otherwise. Live API confirms: `maxuploadsize: 5368709120`. unit GiB; metered on unstated
Price	none stated
Per-object cap	5 GiB/file (5368709120 bytes)
Retention	not stated
Egress	free, unmetered, CDN (upload.wikimedia.org), no auth, Range supported. Wikimedia runs its own edge cache in front of Swift.
Identity	email email — account is username+password+email, no affiliation. Note the nuance: upload itself is in the base `user` group, but `reupload` (overwrite) needs autoconfirmed. So first uploads are open; overwriting an existing file needs account age.
Limits	per-object-cap 5 GiB/file source write-gated review at the community level source 5 GiB/file, 100 MiB without chunking; per-licence/scope review at the community level; thumbnailing capped at a 59 s timeout, 100 MP for GIFs
Binding	per-object-cap the per-object cap sets the shard size and therefore the object count; it bounds the total more tightly than any headline quota
Interface	rest, cli MediaWiki Action API + REST; `action=upload` with `stash`/`filekey` for chunked; `upload_by_url` right for server-side upload
Tooling	mwclient / pywikibot / commons-cli; `upload` and `upload_by_url` are granted to the plain `user` group (verified in siteinfo usergroups), so a brand-new account can already do server-side upload without waiting
Composes with	`upload_by_url` means Commons fetches the file from YOUR url — no local bandwidth at all. Files land at upload.wikimedia.org/wikipedia/commons/<x>/<xy>/<name>, hash-addressed and immutably versioned.
Reliability	The one platform here where a multi-TB batch is explicitly contemplated — Commons asks you to warn sysadmins before 'batch uploads amounting to several TB', which is an acknowledgement that people do it.
Notes	NO TOTAL QUOTA — read verbatim from Commons:Maximum_file_size: 'there is no limit to the total size of uploads by an individual account.' Per-file cap is 5 GiB (5,368,709,120 bytes) via chunked upload, 100 MiB otherwise. Live API confirms: `maxuploadsize: 5368709120`.
Evidence rung	reported One page read moves it to documented.
Checked	2026-10-21 (inherited); stale on 2026-11-04
Parsed numbers	`$.limits[0].value` limit:per-object-cap 5.0 GiB

Evidence (5)

date	source	what it proved
2026-10-21	commons.wikimedia.org/w/api.php?action=query&meta=siteinfo&s	maxuploadsize: 5368709120, generator MediaWiki 1.47.0-wmf.24
2026-10-21	withheld	same API with siprop=usergroups — `user` group holds ['upload','upload_by_url','reupload-own','move-categorypages']
2026-10-21	commons.wikimedia.org/wiki/Commons:Maximum_file_size	'no limit to the total size of uploads by an individual account'; 5 GiB chunked / 100 MiB direct
2026-10-21	commons.wikimedia.org/w/api.php?...siprop=statistics	149,839,243 files on the cluster (scale of the storage layer)
2026-10-21	wikitech.wikimedia.org/wiki/Uploading_large_files	'Our swift backend storage is limited to files smaller than 5 Gb'

Unknown or withheld values (5)

$.evidence[1].url private-url
$.free_tier.notes process
$.per_object_cap.notes not-extracted
$.retention.notes not-extracted
$.retention.stated not-extracted