01
Mercator and the Engineering of a Scalable Web Crawler
Mercator documented the crawler as a distributed systems problem: URL scheduling, duplicate detection, politeness, parsing, extensibility, and persistent state all had to work at Web scale.
↗
Collection