Skip to main content

Web structure and search engine

Short answer

The web is organized as a directed graph of pages connected by hyperlinks, with a bow-tie structure (core SCC, IN, OUT). Search engines exploit this structure: crawlers follow links to discover pages, index content into an inverted index, and use link analysis like PageRank to rank pages; highly linked core pages are found easily.

What is a web? How is it organized? Discuss the working method of search engine with respect to the structure of the web?

What is the web?

The World Wide Web (WWW) is a huge, distributed collection of interlinked documents (web pages, images, videos) stored on web servers around the world and accessed over the Internet using HTTP and URLs. Documents are written in HTML and connected by hyperlinks.

How the web is organized

  • Client-server model: browsers (clients) request pages from web servers.
  • URLs give each resource a unique address; HTTP transfers it; HTML structures it.
  • Hyperlinks connect pages, forming a directed graph (pages are nodes, links are edges).
  • The graph has a bow-tie structure: a strongly connected core (SCC), an IN set that links into the core, an OUT set that is linked from the core, plus tendrils and disconnected parts.
  • Sites are organised in domains, and pages inside a site are arranged in a hierarchy (home page, sections, pages).
  • The surface web is indexed by search engines; the deep web (databases, login pages) is not.
  IN --> [  CORE (SCC)  ] --> OUT
         tendrils / disconnected

Working of a search engine with respect to the web structure

  1. Crawling: the crawler starts from seed URLs and follows hyperlinks, so it uses the link structure of the web to discover pages. Pages in the core are found easily; isolated pages with no in-links are missed.
  2. Indexing: text of every page is processed into an inverted index (word -> pages).
  3. Link analysis: hyperlinks are used to compute importance (PageRank, HITS). Pages with many in-links from important pages rank higher; anchor text describes the target page.
  4. Query processing and ranking: for a query, pages are looked up in the index, scored by content relevance and link-based importance, and returned in order.
  5. Updating: pages are re-crawled periodically to keep the index fresh.

✓ Answer checked by Claude Opus 5.5 (high effort)

Related questions

All Information System chapter 7 questions →