---
title: "Working of web crawlers"
subject: "Web Technology"
chapter: 2
key: "semester_7/web_technology:web-crawler-working"
source: https://www.bctengineer.com/q/semester_7/web_technology/web-crawler-working
site: BCT Engineering
---

# Working of web crawlers

Web Technology · Chapter 2: Web basics

- Asked in: 2076 Chaitra, 8 marks; year not given, 2.5 marks
- Video: [Web Crawlers EXPOSED! How They Really Work](https://www.youtube.com/watch?v=8lXwco6mNK4) — Code potta road
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_7/web_technology/web-crawler-working
- Whole chapter: https://www.bctengineer.com/md/semester_7/web_technology/chapter-2

## Question

Explain the working of web crawlers.

## Short answer

A web crawler starts with seed URLs and uses a URL frontier. It fetches pages via HTTP, parses them to extract links, filters duplicates, and adds new URLs to the frontier while obeying robots.txt and politeness delays for indexing.

## Answer

A **web crawler** (also called spider or bot) is a program that automatically browses the Web, follows hyperlinks and downloads pages. Search engines such as Google (Googlebot) use crawlers to collect pages that are later indexed and searched.

#### Working steps
1. **Seed URLs:** The crawler starts with a list of known URLs (the seed set), placed in a queue called the **URL frontier**.
2. **Fetch:** It takes a URL from the frontier, resolves the domain through DNS and downloads the page with an HTTP request. It first reads the site's `robots.txt` to learn which paths it may visit.
3. **Parse:** The HTML is parsed to extract text, title, metadata and all hyperlinks.
4. **Store:** The page content is saved in a repository for the indexer.
5. **Extract and filter links:** New URLs are made absolute, normalised, and checked against a "seen" set to avoid duplicates and loops.
6. **Add to frontier:** Unseen URLs are added to the frontier, usually ordered by priority (page importance, freshness, domain).
7. **Repeat:** The cycle continues until the frontier is empty or a limit is reached. Pages are revisited periodically to detect updates.

```text
 Seed URLs
    |
    v
+-----------+  fetch  +----------+  parse  +-----------+
| URL       |-------->| Download |-------->| Extract   |
| frontier  |         | page     |         | links+text|
+-----------+         +----------+         +-----------+
    ^                       |                    |
    |  new, unseen URLs     v                    |
    +-----------------  Page repository / Indexer
```

#### Important issues
- **Politeness:** Delay between requests to the same server so it is not overloaded.
- **Robots exclusion:** Obey `robots.txt` and meta robots tags.
- **Duplicate detection:** Use hashes/fingerprints to skip repeated content.
- **Scalability:** Large crawlers run distributed on many machines.
- **Deep web / dynamic pages:** Pages behind forms or logins are hard to crawl.

Uses: search engine indexing, price comparison, web archiving, link checking and data mining.

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. Where does a crawler begin and what does it keep the pending URLs in?
   A. Seed URLs, held in a URL frontier queue
   B. Search results, held in the index
   C. DNS records, held in robots.txt
   D. Hash tables of page content only
   Answer: A. Crawling starts with the seed set placed in the URL frontier. (Key point: Seed and frontier)

2. Why does the crawler read robots.txt before fetching?
   A. To find the IP address
   B. To learn which paths it may visit
   C. To build the index
   D. To detect duplicate pages
   Answer: B. Robots exclusion tells the crawler which parts of the site are allowed. (Key point: robots.txt)

3. Why are extracted links checked against a 'seen' set?
   A. To make them absolute
   B. To encrypt them
   C. To avoid duplicates and loops
   D. To rank pages
   Answer: C. Without checking, the crawler could revisit pages endlessly. (Key point: Seen set)

4. What does crawler politeness mean?
   A. Crawling only polite websites
   B. Ignoring robots.txt
   C. Downloading all pages at once
   D. Delaying between requests to the same server so it is not overloaded
   Answer: D. Politeness limits load on a server. (Key point: Politeness)

## Related in other subjects

- [Web structure and search engine](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Aweb-organization-search-engine-working) — Information System, chapter 7