Tag: search-engines

Mercator documented the crawler as a distributed systems problem: URL scheduling, duplicate detection, politeness, parsing, extensibility, and persistent state all had to work at Web scale.

The inverted index reorganized text around terms and postings rather than documents, giving search engines a scalable way to find the small subset of documents that contain a query term.