web-archive
Common Crawl (CDX index)
A nonprofit project that crawls the open web at petabyte scale and publishes everything it collects, with roughly 2.2 billion pages per crawl and a fresh crawl released regularly. Each crawl has a free, keyless index API that reports whether and when a URL was captured, and the stored page content can then be pulled from the public dataset. It is a series of crawl snapshots rather than an on-demand archive, so it suits checking how a site looked in recent crawl windows or doing bulk web research.
Warum es nützlich ist & wie es funktioniert
Common Crawl is the best resource when you want to know whether a URL was captured in a recent broad crawl of the open web, or when you are doing bulk research across many sites at once. Each monthly crawl publishes a free, keyless index API: supply a URL and it returns whether and when that address was captured, along with a pointer to the stored content in the public dataset. Because it is a series of periodic crawl snapshots rather than an on-demand archive, it suits checking a site's state in recent months rather than retrieving arbitrary historical versions.
Was drinsteckt
Each crawl covers roughly 2.2 billion web pages. The April 2026 crawl (CC-MAIN-2026-17) contains approximately 2.19 billion pages across 379 terabytes of uncompressed content. The cumulative archive across all crawls since 2008 spans over 300 billion pages.
API-Zugang
https://index.commoncrawl.org/CC-MAIN-2026-25-index?url= <url>&output=json ; collection list https://index.commoncrawl.org/collinfo.json ; WARC on s3://commoncrawl/ and https://data.commoncrawl.org/
Zugang
Programmatic API access (a key may be required, see the API tag).