This collection contains an extracted web archive corpus of 0.8+ million corporate websites (from an original list of ~0.98 websites) extracted from the archive.org web archive, covering the period 1996 to early 2017. This corpus was originally created as a collaboration between the Internet Archive and a group at Dartmouth University, but it may be useful to other researchers.
Updated or more detailed information may exist at:
Corpus Statistics
- approximately 840,000 domains
- more than 500,000,000 unique URLs
- more than 1,600 WARC files and 26,000 ARC files
- more than 3.25 TB compressed
TODO: link to future publications TODO: DataCite DOI
How to Download
This large corpus is split into several hundred distinct "items" on archive.org.
We recommend using the internetarchive python utility (aka, "ia") for bulk downloads. See also:
WARC and ARC downloads can be verified using the unified manifest files at:
Additional manifest and derived information (including CDX files) exist at:
Methodology
TODO: document how this dataset was generated
