Crawling
Crawl, search, reader and unblocker APIs for reading the web at scale.
49 records: 32 usable, 6 trial, 1 paid only. 10 rejected or dead.
Best per need: Free crawling, search and reader APIs
Usable 32
search api 9
- Exa caveatemailcardDrequest-rateUSD 10 in credits on signup, reset to USD 10 on the first of each month, no payment method; a one-time USD 10 bonus after onboarding
- Keenable caveatemailno cardDrequest-rate100,000 requests a month
- SerpApi caveatemailno cardDrequest-rate250 searches a month on the Free plan, 50 an hour
- Tavily caveatemailno cardDrequest-rate1,000 API credits a month, no credit card, reset on the first of the month
- You.com caveatanonymousno cardDrequest-rate100 Web Search API queries a day through the hosted MCP route, with no API key, no account and no credit card
- Firecrawl keyless route caveatanonymousno cardRrequest-ratea keyless route to the Research Index only
- Parallel caveatcardcardRrequest-rate5,000 requests a month, card on file
- Serper caveatemailno cardRrequest-rate2,500 queries on signup, one-time, no credit card
- Tavily keyless route caveatanonymousno cardRrequest-ratea shared public route with no key
archive 5
- Internet Archive Wayback Machine ★ qualifiesanonymousno cardMper-object-capkeyless listing, closest and replay of existing captures; Save Page Now needs keys
- Common Crawl caveatanonymousno cardMrequest-ratekeyless index and WARC access
- Cloudflare URL Scanner caveatemailno cardDretention-clampedfree with a Cloudflare account; scans of the exact URL in the last 30 days
- Arquivo.pt caveatanonymousno cardRrequest-ratekeyless CDX and replay of the Portuguese web archive
- urlscan.io caveatanonymousno cardRrequest-rate1,000 unlisted and 5,000 public scans a day, 1,000 searches a day, 10,000 result requests a day
reader and extractor api 4
- Jina Reader ★ qualifiesanonymousno cardMper-ip-shapingr.jina.ai is free at 20 requests a minute with no key; a free API key raises it to 500 RPM and carries 10M tokens
- Context.dev ★ qualifiesemailno cardDrequest-rate1,000 credits a month, no credit card, 10 requests a minute
- Diffbot caveatemailno cardDrequest-rate10,000 credits a month on the Free plan, 5 requests a minute, no credit card
- Scrape.do caveatemailno cardRrequest-rate1,000 successful calls a month
fetch relay and proxy 4
- Crawlbase caveatemailno cardDper-object-capup to 5,000 successful requests free, no credit card
- ScraperAPI caveatemailno cardDper-object-cap1,000 API credits on signup, 5 concurrent connections
- Scrapfly caveatemailno cardDper-object-cap1,000 API credits on signup, no credit card, 5 concurrent
- ZenRows caveatemailno cardDper-object-cap5,000 credits a month, no card, 5 concurrent requests, monthly refresh with no rollover
crawl api 3
- Firecrawl ★ qualifiesemailno cardDrequest-rate500 searches or 1,000 pages scraped a month on the Free plan, 2 concurrent requests, no card
- Apify caveatemailno cardDper-object-capUSD 5 of platform usage a month on the Free plan, 5 concurrent runs
- Crawl4AI caveatemailno cardRper-object-cap1,000 credits a month on the hosted cloud; the pricing page also declares 2,500
other 7
- OpenAlex ★ qualifiesemailcardDrequest-rateUSD 1 of API usage a day with a free key; keyless requests at 10 a second
- Bright Data caveatemailno cardDrequest-rate5,000 records a month on the Free Tier, no credit card
- Browserless caveatemailno cardDretention-clamped1,000 units a month, no card, 2 concurrent browsers, 2 minutes maximum session
- ScrapingAnt caveatemailno cardDper-object-cap10,000 credits a month, no credit card, refreshed monthly
- TinyFish caveatemailno cardDrequest-rateSearch free at 30 requests a minute and 500 an hour; Fetch free at 150 URLs a minute and 1,000 URLs a day
- Cloudflare Browser Rendering caveatemailno cardRrequest-rateWorkers Free: 600 browser-seconds a day, or 10 browser-minutes; 3 concurrent browsers
- Cloudflare Workers AI caveatemailno cardRrequest-rate10,000 Neurons a day per account on Workers Free
Trial 6
- Browserbase trialemailno cardDrequest-rateFree plan: 1 browser hour, 1,000 Search calls, 1,000 Fetch calls, 3 concurrent browsers
- Hyperbrowser trialemailno cardDretention-clamped5,000 credits on signup, 1 concurrent session, 7 days of data retention
- Octen trialemailno cardDrequest-rateUSD 5 in free balance on signup
- Anchor Browser trialemailno cardRrequest-rateUSD 5 of credits a month per the docs, or 5 credits a month per the pricing card; the two disagree
- Notte trialemailno cardRretention-clampedUSD 10 of credits, one time, no expiry
- Steel.dev trialemailno cardRretention-clampedUSD 30 of credits, one time, valid 90 days
Paid only 1
- Perplexity paidcardcardDrequest-ratenone: the Search API is billed per successful request from the first one