--- title: "Web structure and search engine" subject: "Information System" chapter: 7 key: "semester_8/information_system:web-organization-search-engine-working" source: https://www.bctengineer.com/q/semester_8/information_system/web-organization-search-engine-working site: BCT Engineering --- # Web structure and search engine Information System · Chapter 7: Web based information system and navigation - Asked in: 2071 Bhadra, 5 marks - Video: [How Search Really Works](https://www.youtube.com/watch?v=TByRaraQqW4) — ByteByteGo - Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14 - Page: https://www.bctengineer.com/q/semester_8/information_system/web-organization-search-engine-working - Whole chapter: https://www.bctengineer.com/md/semester_8/information_system/chapter-7 ## Question What is a web? How is it organized? Discuss the working method of search engine with respect to the structure of the web? ## Short answer The web is organized as a directed graph of pages connected by hyperlinks, with a bow-tie structure (core SCC, IN, OUT). Search engines exploit this structure: crawlers follow links to discover pages, index content into an inverted index, and use link analysis like PageRank to rank pages; highly linked core pages are found easily. ## Answer #### What is the web? The **World Wide Web (WWW)** is a huge, distributed collection of interlinked documents (web pages, images, videos) stored on web servers around the world and accessed over the Internet using HTTP and URLs. Documents are written in HTML and connected by hyperlinks. #### How the web is organized - **Client-server model:** browsers (clients) request pages from web servers. - **URLs** give each resource a unique address; **HTTP** transfers it; **HTML** structures it. - **Hyperlinks** connect pages, forming a **directed graph** (pages are nodes, links are edges). - The graph has a **bow-tie structure**: a strongly connected core (SCC), an IN set that links into the core, an OUT set that is linked from the core, plus tendrils and disconnected parts. - Sites are organised in domains, and pages inside a site are arranged in a hierarchy (home page, sections, pages). - The **surface web** is indexed by search engines; the **deep web** (databases, login pages) is not. ```text IN --> [ CORE (SCC) ] --> OUT tendrils / disconnected ``` #### Working of a search engine with respect to the web structure 1. **Crawling:** the crawler starts from seed URLs and follows hyperlinks, so it uses the link structure of the web to discover pages. Pages in the core are found easily; isolated pages with no in-links are missed. 2. **Indexing:** text of every page is processed into an inverted index (word -> pages). 3. **Link analysis:** hyperlinks are used to compute importance (PageRank, HITS). Pages with many in-links from important pages rank higher; anchor text describes the target page. 4. **Query processing and ranking:** for a query, pages are looked up in the index, scored by content relevance and link-based importance, and returned in order. 5. **Updating:** pages are re-crawled periodically to keep the index fresh. ## Check yourself The answer's key points as multiple-choice questions, with the correct option and why. 1. How is the web viewed structurally? A. As a directed graph with pages as nodes and hyperlinks as edges B. As a flat list of files C. As a single tree with one root D. As a ring of servers Answer: A. Hyperlinks connect pages to form a directed graph. (Key point: Web as graph) 2. What are the parts of the bow-tie structure? A. Head, body and tail only B. Client, server and router C. Surface, deep and dark only D. A strongly connected core, IN set, OUT set, tendrils and disconnected parts Answer: D. IN links into the core, OUT is linked from the core. (Key point: Bow-tie) 3. Why may a crawler miss some pages? A. Pages in the core are hidden B. All pages are blocked C. Isolated pages with no in-links cannot be reached by following links D. Crawlers read only images Answer: C. Crawlers discover pages by following links. (Key point: Crawling and structure) 4. What is the deep web? A. The pages with most links B. Content such as databases and login pages not indexed by search engines C. The home pages of sites D. Pages stored in the cache only Answer: B. The surface web is indexed; the deep web is not. (Key point: Deep web) ## Related in Information System - [Link analysis and search engine architecture](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Alink-analysis-search-engine-architecture) — Information System, chapter 7 - [Web mining with structure mining](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Aweb-mining-structure-mining) — Information System, chapter 7 ## Related in other subjects - [Working of web crawlers](https://www.bctengineer.com/question.md?key=semester_7%2Fweb_technology%3Aweb-crawler-working) — Web Technology, chapter 2 - [Four-stroke SI engine working and components](https://www.bctengineer.com/question.md?key=semester_6%2Finternal_combustion_engines%3Afour-stroke-si-engine-working-components) — Internal Combustion Engines, chapter 2 - [Web structure/content/usage mining; privacy](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Aweb-structure-content-usage-privacy) — Data Mining, chapter 7 - [Web mining structure and challenges](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Aweb-mining-structure-challenges) — Data Mining, chapter 7