{"eid":"wikimedia","name":"Wikimedia Downloads","domain":"wikimedia.org","resolved_by":"domain","topics":["debrid","storage"],"n_records":2,"best_vorder":1,"review":null,"aliases":["Wikimedia Commons","Wikimedia Downloads","Wikimedia Downloads (dumps.wikimedia.org)","wikimedia.org"],"records":[{"rid":"debrid/wikimedia-dumps","card":{"rid":"debrid/wikimedia-dumps","eid":"wikimedia","topic":"debrid","slug":"wikimedia-dumps","provider":"Wikimedia Downloads (dumps.wikimedia.org)","url":"https://dumps.wikimedia.org/","domain":"wikimedia.org","category":"encyclopaedia corpus (monthly XML dumps, HTTP mirrors)","tier":"free","verdict":"qualifies-with-caveat","vorder":1,"grp":"usable","confidence":"measured","binding":"per-ip-shaping","identity_kind":"anonymous","identity_basis":"stated","needs_card":0,"needs_phone":0,"needs_id":0,"needs_institution":0,"country":null,"crypto":null,"interfaces":"rest,rsync,cli","egress_metered":0,"per_object_bytes":null,"retention_days":null,"verified_at":"2026-10-23T03:24:34.450Z","verified_basis":"live","checked_on":"2026-10-24","stale_on":"2026-11-23","n_evidence":4,"usd_month":null,"usd_per_tb_month":0.0,"summary":"Complete public wiki dumps, monthly, free and anonymous. 'Downloads are also rate limited and capped at 3 connections per-IP.' The English Wikipedia article ...","duplicate_of":null,"n_unknown":0},"record":{"topic":"debrid","slug":"wikimedia-dumps","provider":"Wikimedia Downloads (dumps.wikimedia.org)","url":"https://dumps.wikimedia.org/","category":"encyclopaedia corpus (monthly XML dumps, HTTP mirrors)","tier":"free","free_tier":{"stated":"Complete public wiki dumps, monthly, free and anonymous. 'Downloads are also rate limited and capped at 3 connections per-IP.' The English Wikipedia article dump measured here is 25,800,511,704 bytes.","unit":"unlimited","metered_on":"none","source_url":"https://dumps.wikimedia.org/","read_on":"2026-10-23","notes":"The dumps are being reorganised: the XML dumps are 'deprecated and will be sunset in January 2027' in favour of MediaWiki Content File Exports."},"per_object_cap":{"stated":"not applicable: one compressed file per dump. enwiki-latest-pages-articles.xml.bz2 is 25,800,511,704 bytes; ranged reads are supported (accept-ranges: bytes).","bytes":null,"source_url":"https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2","notes":""},"retention":{"stated":"monthly snapshots are kept; 'latest' is replaced each run and older dated directories persist for a period the dump index does not quantify","days":null,"source_url":"https://dumps.wikimedia.org/","notes":""},"egress":{"terms":"Free, but conditioned on the User-Agent policy: 'If you are reading this on Wikimedia servers, please note that we are now enforcing the Wikimedia Foundation User-Agent Policy for dumps downloads. Downloads are also rate limited and capped at 3 connections per-IP ... Clients that do not respect the user-agent policy or who try to evade these limits may be blocked.'","metered":false,"free_allowance":"no byte allowance published","price_per_tb":null,"source_url":"https://dumps.wikimedia.org/"},"interface":["rest","rsync","cli"],"interface_detail":"HTTPS per file with Range support; rsync mirrors are listed on meta.wikimedia.org; the dump index at /backup-index.html gives every file with size and checksum.","endpoint":"https://dumps.wikimedia.org/enwiki/latest/","tooling":"curl/wget with a descriptive User-Agent, rsync against a mirror, and the checksum files that ship beside each dump.","chaining":"A fixed encyclopaedia corpus to join against Common Crawl's web text; the 24 GiB article dump is small enough to hold beside a dataset snapshot, and the mirror list is the escape hatch when the per-IP cap binds.","limits":[{"kind":"per-ip-shaping","value":"capped at 3 connections per IP, and rate limited; measured here at 1.661-2.617 MiB/s over one connection (three runs) and 2.647-7.649 MiB/s total over eight connections (three runs), so parallelism helped up to 3.6x and the cap did not refuse the eighth connection","source":"https://dumps.wikimedia.org/"}],"limits_note":"The User-Agent policy is enforced: a client without a descriptive agent may be blocked, and evasion attempts are named as a blocking reason on the front page.","binding":"per-ip-shaping","binding_note":"The documented 3-connection cap is a connection count, not a byte rate: eight parallel range reads all returned 206 and the best of them reached 7.649 MiB/s, which is about 645 GiB/day.","identity_cost":{"kind":"anonymous","detail":"No account and no key; a descriptive User-Agent is the only requirement, and it is not a credential.","country":null},"reliability_note":"The dump format is changing: XML dumps sunset in January 2027, so a pipeline built on pages-articles.xml has a dated migration to the Content File Exports format.","confidence":"measured","confidence_note":"The file size came from a HEAD and the rate from a 64 MiB ranged read, three runs at each concurrency. All eight parallel requests returned 206, so the documented 3-connection cap did not refuse them at the transport level - whether it is enforced by rate, by UA or only at peak is not established.","evidence":[{"url":"https://dumps.wikimedia.org/","date":"2026-10-23","proved":"'Downloads are also rate limited and capped at 3 connections per-IP', User-Agent policy enforcement, and the January 2027 XML sunset notice"},{"url":"probe:https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2 --head","date":"2026-10-23","proved":"200, content-length 25,800,511,704, accept-ranges: bytes, last-modified 2026-10-08"},{"url":"probe:https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2 --range 0-67108863","date":"2026-10-23","proved":"64 MiB ranges at one connection: 2.617, 2.084 and 1.661 MiB/s across three runs, 206 with the range honoured; sha256 of the first range 3a74bf06ec319f7d702771523ec099593e7569f00b5bddfdbe3e76ddbc914ea7"},{"url":"probe:https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2 --concurrency 8 --shard-bytes 67108864","date":"2026-10-23","proved":"eight parallel 206s totalling 2.647, 4.748 and 7.649 MiB/s against same-run one-connection baselines of 1.317, 2.918 and 2.337 MiB/s - parallelism helped, and no request was refused"}],"verdict":"qualifies-with-caveat","verified_at":"2026-10-23T03:24:34.450Z","price_per_tb_month":"USD 0","notes":"Caveat is the 2027 format change and the 3-connection rule. A single connection gives about 178 GiB/day at the mean, and the best eight-connection run reached 645 GiB/day - which the published cap says a client should not do. The whole English Wikipedia article dump is one day's single-connection allowance.","extras":{"enwiki_articles_xml_bytes":25800511704,"one_conn_mib_s":[2.617,2.084,1.661],"eight_conn_mib_s":[2.647,4.748,7.649],"connection_cap":3,"range_sha256":"3a74bf06ec319f7d702771523ec099593e7569f00b5bddfdbe3e76ddbc914ea7","xml_sunset":"2027-01"}},"claims":[{"path":"$.free_tier.stated","kind":"free-quota","num":null,"unit":null,"per":null,"currency":null,"usd":null,"fx_date":null,"parse":"unparsed"},{"path":"$.limits[0].value","kind":"limit:per-ip-shaping","num":2.617,"unit":"MiB","per":null,"currency":null,"usd":null,"fx_date":null,"parse":"ok"},{"path":"$.price_per_tb_month","kind":"price-per-tb","num":0.0,"unit":"TB","per":"month","currency":"USD","usd":0.0,"fx_date":null,"parse":"ok"}],"unknown":[],"history":[],"drift":[]},{"rid":"storage/wikimedia-commons","card":{"rid":"storage/wikimedia-commons","eid":"wikimedia","topic":"storage","slug":"wikimedia-commons","provider":"Wikimedia Commons","url":"https://commons.wikimedia.org","domain":"wikimedia.org","category":"open media repository (institutional-grade, Wikimedia Foundation)","tier":null,"verdict":"qualifies-with-caveat","vorder":1,"grp":"usable","confidence":"reported","binding":"per-object-cap","identity_kind":"email","identity_basis":"stated","needs_card":0,"needs_phone":0,"needs_id":0,"needs_institution":0,"country":null,"crypto":null,"interfaces":"rest,cli","egress_metered":0,"per_object_bytes":5368709120,"retention_days":null,"verified_at":"2026-10-21T00:00:00.000Z","verified_basis":"inherited","checked_on":"2026-10-21","stale_on":"2026-11-04","n_evidence":5,"usd_month":null,"usd_per_tb_month":null,"summary":"NO TOTAL QUOTA — read verbatim from Commons:Maximum_file_size: 'there is no limit to the *total* size of uploads by an individual account.' Per-file cap is 5...","duplicate_of":null,"n_unknown":5},"record":{"topic":"storage","slug":"wikimedia-commons","provider":"Wikimedia Commons","url":"https://commons.wikimedia.org","category":"open media repository (institutional-grade, Wikimedia Foundation)","free_tier":{"stated":"NO TOTAL QUOTA — read verbatim from Commons:Maximum_file_size: 'there is no limit to the *total* size of uploads by an individual account.' Per-file cap is 5 GiB (5,368,709,120 bytes) via chunked upload, 100 MiB otherwise. Live API confirms: `maxuploadsize: 5368709120`.","unit":"GiB","metered_on":"unstated","source_url":"https://commons.wikimedia.org/wiki/Commons:Maximum_file_size","read_on":"2026-10-21","notes":null},"per_object_cap":{"stated":"5 GiB/file","bytes":5368709120,"source_url":"https://commons.wikimedia.org/wiki/Commons:Maximum_file_size","notes":"not stated"},"retention":{"stated":"not stated","source_url":"https://commons.wikimedia.org/wiki/Commons:Maximum_file_size","notes":"not stated"},"egress":{"terms":"free, unmetered, CDN (upload.wikimedia.org), no auth, Range supported. Wikimedia runs its own edge cache in front of Swift.","metered":false},"interface":["rest","cli"],"interface_detail":"MediaWiki Action API + REST; `action=upload` with `stash`/`filekey` for chunked; `upload_by_url` right for server-side upload","tooling":"mwclient / pywikibot / commons-cli; `upload` and `upload_by_url` are granted to the plain `user` group (verified in siteinfo usergroups), so a brand-new account can already do server-side upload without waiting","chaining":"`upload_by_url` means Commons fetches the file from YOUR url — no local bandwidth at all. Files land at upload.wikimedia.org/wikipedia/commons/<x>/<xy>/<name>, hash-addressed and immutably versioned.","limits":[{"kind":"per-object-cap","value":"5 GiB/file","source":"https://commons.wikimedia.org/wiki/Commons:Maximum_file_size"},{"kind":"write-gated","value":"review at the community level","source":"https://commons.wikimedia.org/wiki/Commons:Maximum_file_size"}],"limits_note":"5 GiB/file, 100 MiB without chunking; per-licence/scope review at the community level; thumbnailing capped at a 59 s timeout, 100 MP for GIFs","identity_cost":{"kind":"email","detail":"email — account is username+password+email, no affiliation. Note the nuance: upload itself is in the base `user` group, but `reupload` (overwrite) needs autoconfirmed. So first uploads are open; overwriting an existing file needs account age.","country":null},"reliability_note":"The one platform here where a multi-TB batch is explicitly contemplated — Commons asks you to warn sysadmins before 'batch uploads amounting to several TB', which is an acknowledgement that people do it.","confidence":"reported","confidence_note":"One page read moves it to documented.","evidence":[{"url":"https://commons.wikimedia.org/w/api.php?action=query&meta=siteinfo&siprop=general&format=json","date":"2026-10-21","proved":"maxuploadsize: 5368709120, generator MediaWiki 1.47.0-wmf.24"},{"url":null,"date":"2026-10-21","proved":"same API with siprop=usergroups — `user` group holds ['upload','upload_by_url','reupload-own','move-categorypages']"},{"url":"https://commons.wikimedia.org/wiki/Commons:Maximum_file_size","date":"2026-10-21","proved":"'no limit to the *total* size of uploads by an individual account'; 5 GiB chunked / 100 MiB direct"},{"url":"https://commons.wikimedia.org/w/api.php?...siprop=statistics","date":"2026-10-21","proved":"149,839,243 files on the cluster (scale of the storage layer)"},{"url":"https://wikitech.wikimedia.org/wiki/Uploading_large_files","date":"2026-10-21","proved":"'Our swift backend storage is limited to files smaller than 5 Gb'"}],"verdict":"qualifies-with-caveat","verified_at":"2026-10-21T00:00:00.000Z","notes":"NO TOTAL QUOTA — read verbatim from Commons:Maximum_file_size: 'there is no limit to the *total* size of uploads by an individual account.' Per-file cap is 5 GiB (5,368,709,120 bytes) via chunked upload, 100 MiB otherwise. Live API confirms: `maxuploadsize: 5368709120`.","extras":{},"binding":"per-object-cap","binding_note":"the per-object cap sets the shard size and therefore the object count; it bounds the total more tightly than any headline quota"},"claims":[{"path":"$.free_tier.stated","kind":"free-quota","num":5.0,"unit":"GiB","per":null,"currency":null,"usd":null,"fx_date":null,"parse":"ok"},{"path":"$.limits[0].value","kind":"limit:per-object-cap","num":5.0,"unit":"GiB","per":null,"currency":null,"usd":null,"fx_date":null,"parse":"ok"},{"path":"$.limits[1].value","kind":"limit:write-gated","num":null,"unit":null,"per":null,"currency":null,"usd":null,"fx_date":null,"parse":"unparsed"}],"unknown":[["$.evidence[1].url","private-url"],["$.free_tier.notes","process"],["$.per_object_cap.notes","not-extracted"],["$.retention.notes","not-extracted"],["$.retention.stated","not-extracted"]],"history":[],"drift":[]}]}