web-archive

Common Crawl (CDX index)

A nonprofit project that crawls the open web at petabyte scale and publishes everything it collects, with roughly 2.2 billion pages per crawl and a fresh crawl released regularly. Each crawl has a free, keyless index API that reports whether and when a URL was captured, and the stored page content can then be pulled from the public dataset. It is a series of crawl snapshots rather than an on-demand archive, so it suits checking how a site looked in recent crawl windows or doing bulk web research.

Web pages / sites API

Search this archive

Why it’s useful & how it works

Common Crawl is the best resource when you want to know whether a URL was captured in a recent broad crawl of the open web, or when you are doing bulk research across many sites at once. Each monthly crawl publishes a free, keyless index API: supply a URL and it returns whether and when that address was captured, along with a pointer to the stored content in the public dataset. Because it is a series of periodic crawl snapshots rather than an on-demand archive, it suits checking a site's state in recent months rather than retrieving arbitrary historical versions.

What’s inside

Each crawl covers roughly 2.2 billion web pages. The April 2026 crawl (CC-MAIN-2026-17) contains approximately 2.19 billion pages across 379 terabytes of uncompressed content. The cumulative archive across all crawls since 2008 spans over 300 billion pages.

API access

https://index.commoncrawl.org/CC-MAIN-2026-25-index?url= <url>&output=json ; collection list https://index.commoncrawl.org/collinfo.json ; WARC on s3://commoncrawl/ and https://data.commoncrawl.org/

What we measured

Our own probes, not the archive’s own claims. Re-run periodically; every reading below is dated.

Reachability

Direct request: Responded HTTP 200 827 ms
Through a datacenter proxy: Responded HTTP 200 869 ms
API, direct: Responded HTTP 200 487 ms
API, through a proxy: Responded HTTP 200 784 ms

Did a real search return anything?

Returned results · 20 results for the sample query · median 1.2 s · stable across runs

Sample query: https://www.example.com/

Benchmark coverage

Found 7 of 10 test items (70%) in the Web pages / sites cluster. Median response 434 ms.

By how well known the item is: 3/3 well-known, 2/4 mid, 2/3 obscure .

Reachability measured 2026-08-22. Search checked 2026-08-22. Benchmark run 2026-08-22.