Common Crawl

commoncrawl.org · Crawling, Storage · 2 records · JSON

Common Crawl Crawlingcaveatanonymousno cardMrequest-rate
Common Crawl (the petabyte corpus — read side only) Storagefails: quotainstitution?card?R

Common Crawl Crawling

caveatanonymousno cardMrequest-rate

commoncrawl.org/ · archive · crawler-scraper/commoncrawl · JSON · compare

Free tier	keyless index and WARC access unit requests; metered on request-count
Price	none stated
Per-object cap	not applicable: the unit is a WARC record
Retention	each crawl keeps its own index; up to 12 are asked
Egress	not applicable: the archived body, read from a WARC
Identity	anonymous keyless
Limits	request-rate about 1 request a second on the index, per the platform research report request-rate a crawl index call is bounded at 5 s by the broker; the listing ends at 8 s
Binding	request-rate
Interface	rest Index API per crawl on index.commoncrawl.org; WARC files on data.commoncrawl.org
Tooling	not stated
Composes with	The last archive the broker asks, and the only one that can answer more true on an otherwise empty listing
Reliability	504 to a direct index call; 9 consecutive probe timeouts as of 02:50Z 2026-10-22
Notes	Degraded, not dead, and the distinction is the finding: the index answered 504 to a direct call and has failed nine broker probes in a row. A listing can return unknown for every crawl it could not read, so an empty answer is not evidence of absence.
Evidence rung	measured
Checked	2026-10-22 (live); stale on 2026-12-21
Parsed numbers	`$.limits[0].value` limit:request-rate 1.0 request

Evidence (2)

date	source	what it proved
2026-10-22	index.commoncrawl.org/CC-MAIN-2026-38-index?url=example.com&	504 Gateway Time-out after 11.3 s from nginx: the index did not answer
2026-10-22	withheld	state failing, 9 consecutive probe timeouts, the only provider not ok

Unknown or withheld values (3)

$.evidence[1].url private-url
$.limits[1].source private-url
$.tooling not-stated

Common Crawl (the petabyte corpus — read side only) Storage

fails: quotainstitution?card?R

commoncrawl.org/get-started · academic · storage/common-crawl · JSON · compare

Free tier	Not a hosting service: ~300 billion web pages across ~100+ monthly crawls, multi-petabyte, free to READ and free to re-host yourself. There is NO path to host your own data with Common Crawl — confirmed, it publishes its own crawls only. unit mixed; metered on unstated
Price	none stated
Per-object cap	not stated
Retention	not stated
Egress	Free and anonymous over HTTPS: 'The data is accessible via the https://data.commoncrawl.org/[...] URL scheme. There is no need to create an AWS account in order to access the data using this method.' S3 access (s3://commoncrawl) requires an authenticated AWS account. New: selected crawls are mirrored into a Hugging Face Storage Bucket with CDN pre-warming.
Identity	institution (placeholder) inferred from the category 'academic'; the source record states no identity cost, so this is a class assumption, not evidence
Binding	none no constraint of a named kind was recorded; the verdict rests on the quota sentence rather than on a named limit
Interface	s3, cli s3
Tooling	wget/curl against data.commoncrawl.org, aws-cli against s3://commoncrawl (auth), the pycrawl / cc-index-table Athena tooling, and now `hf buckets cp hf://buckets/commoncrawl/commoncrawl/...`.
Composes with	Read-side: any amount of free inbound bandwidth for corpus you would otherwise have to build. The useful pattern for a user is the reverse of storage — Common Crawl is the SOURCE that makes a 200 GB Kaggle/generic pipeline interesting without owning a petabyte.
Reliability	Extremely stable (running since 2008, sponsored by AWS), immutable crawl archives, no access throttling observed on the HTTPS path.
Notes	Not a hosting service: ~300 billion web pages across ~100+ monthly crawls, multi-petabyte, free to READ and free to re-host yourself. There is NO path to host your own data with Common Crawl — confirmed, it publishes its own crawls only.
Evidence rung	reported One page read moves it to documented.
Checked	2026-10-21 (inherited); stale on 2026-11-04

Evidence (3)

date	source	what it proved
2026-10-21	commoncrawl.org/get-started	'There is no need to create an AWS account in order to access the data using this method'; 'access to data from the Amazon cloud using the S3 API is only allowed for authenticated users'
2026-10-21	data.commoncrawl.org/crawl-data/CC-MAIN-2026-17/warc.paths.g	probe: GET -> 200, 201,605 bytes, gzip (anonymous listing manifest)
2026-10-21	commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2026-17/warc	probe: GET -> 403 (S3 path is auth-only)

More fields (1)

{
 "dead_end": {
  "failure_class": "quota",
  "why": "Not a hosting service: ~300 billion web pages across ~100+ monthly crawls, multi-petabyte, free to READ and free to re-host yourself. There is NO path to host your own data with Common Crawl — confirmed, it publishes its own crawls only.",
  "retry_when": null,
  "first_recorded": "2026-10-21"
 }
}

Unknown or withheld values (7)

$.extras.dead_end.retry_when process
$.free_tier.notes process
$.identity_cost.kind identity-defaulted
$.per_object_cap.notes not-extracted
$.per_object_cap.stated not-extracted
$.retention.notes not-extracted
$.retention.stated not-extracted