Working of web crawlers

A web crawler (also called spider or bot) is a program that automatically browses the Web, follows hyperlinks and downloads pages. Search engines such as Google (Googlebot) use crawlers to collect pages that are later indexed and searched.

Working steps

1. Seed URLs: The crawler starts with a list of known URLs (the seed set), placed in a queue called the URL frontier.
2. Fetch: It takes a URL from the frontier, resolves the domain through DNS and downloads the page with an HTTP request. It first reads the site's robots.txt to learn which paths it may visit.
3. Parse: The HTML is parsed to extract text, title, metadata and all hyperlinks.
4. Store: The page content is saved in a repository for the indexer.
5. Extract and filter links: New URLs are made absolute, normalised, and checked against a "seen" set to avoid duplicates and loops.
6. Add to frontier: Unseen URLs are added to the frontier, usually ordered by priority (page importance, freshness, domain).
7. Repeat: The cycle continues until the frontier is empty or a limit is reached. Pages are revisited periodically to detect updates.

 Seed URLs
    |
    v
+-----------+  fetch  +----------+  parse  +-----------+
| URL       |-------->| Download |-------->| Extract   |
| frontier  |         | page     |         | links+text|
+-----------+         +----------+         +-----------+
    ^                       |                    |
    |  new, unseen URLs     v                    |
    +-----------------  Page repository / Indexer

Important issues

• Politeness: Delay between requests to the same server so it is not overloaded.
• Robots exclusion: Obey robots.txt and meta robots tags.
• Duplicate detection: Use hashes/fingerprints to skip repeated content.
• Scalability: Large crawlers run distributed on many machines.
• Deep web / dynamic pages: Pages behind forms or logins are hard to crawl.

Uses: search engine indexing, price comparison, web archiving, link checking and data mining.