--- title: "Components of indexing and searching" subject: "Big Data Technologies" chapter: 5 key: "semester_8/big_data_technologies:indexing-searching-components" source: https://www.bctengineer.com/q/semester_8/big_data_technologies/indexing-searching-components site: BCT Engineering --- # Components of indexing and searching Big Data Technologies · Chapter 5: Searching and Indexing Big Data - Asked in: 2074 Bhadra, 8 marks; 2073 Magh, 10 marks (asked 2 times) - Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14 - Page: https://www.bctengineer.com/q/semester_8/big_data_technologies/indexing-searching-components - Whole chapter: https://www.bctengineer.com/md/semester_8/big_data_technologies/chapter-5 ## Question Explain the components of Indexing and searching. (Describe the typical components involved in a search application.) ## Short answer Indexing components acquire raw content, build documents with fields, analyze text (tokenize, lower-case, remove stop words, stem), and store terms in an inverted index. Searching components include a query UI, parsing/analyzing the query, looking up terms in the inverted index, ranking hits by relevance (e.g., TF-IDF), and rendering results with snippets and highlighting. ## Answer A search application lets users find relevant documents in a large collection quickly. Instead of scanning every document, it builds an **index** in advance and answers queries from that index. The typical components are as follows. ```text Documents -> Acquire -> Build Document -> Analyze -> Index | User -> Query UI -> Build Query -> Run Query -> Render Results ``` #### Indexing components 1. **Acquire raw content:** collect the data to be searched from files, web pages (a crawler), databases, emails, etc. Tools: Nutch, Solr DIH, custom crawlers. 2. **Build documents:** convert raw content into units called *documents*, each having *fields* (title, body, author, date). Text is extracted from PDF, HTML, Word etc. Fields can be boosted by importance. 3. **Analyze documents:** the text is broken into **tokens** by an analyzer: tokenizing, lower-casing, stop-word removal (the, is, of), stemming (running -> run), synonyms. 4. **Index documents:** the tokens are stored in an **inverted index**, which maps each term to the list of documents (and positions) containing it. It is written to disk by an index writer. Example inverted index: | Term | Postings (doc ids) | |---|---| | big | 1, 3 | | data | 1, 2, 3 | | hadoop | 2 | #### Searching components 5. **Search user interface:** the box where the user enters a query, and the page that shows results (with facets, paging, highlighting). It should be simple and fast. 6. **Build query:** the user's text is parsed by a **query parser** into a structured query (terms, phrases, Boolean AND/OR/NOT, wildcard, ranges). The same analyzer is applied to query text. 7. **Run query (search query):** the index searcher looks up the terms in the inverted index, finds the matching documents, and **scores/ranks** them (TF-IDF, BM25) by relevance. 8. **Render results:** the top-ranked documents are shown with title, snippet, highlighted terms, and links, with sorting and pagination. #### Administration components - **Index management:** update, delete, merge segments, optimise, replicate and back up the index. - **Analytics and monitoring:** logs, query statistics, to improve relevance. Examples of platforms: Apache Lucene (library), Solr and Elasticsearch (servers built on Lucene). ## Check yourself The answer's key points as multiple-choice questions, with the correct option and why. 1. Why does a search application build an index in advance? A. To store a backup copy of the documents B. To compress documents to save disk space C. To answer queries from the index instead of scanning every document D. To remove the need for a query parser Answer: C. The index lets queries be answered quickly without scanning every document in the collection. (Key point: Purpose of an index) 2. What happens in the 'analyze documents' step of indexing? A. Raw files are crawled from web pages B. Text is broken into tokens, lower-cased, stop words removed, stemmed, and synonyms handled C. Results are ranked by relevance D. Users type queries into a search box Answer: B. The analyzer turns field text into tokens using tokenizing, lower-casing, stop-word removal, stemming and synonyms. (Key point: Analyze documents step) 3. What does an inverted index store? A. For each document, the list of words it has B. For each user, the queries they ran C. For each field, its font and layout D. For each term, the list of documents (and positions) that contain it Answer: D. An inverted index maps terms to documents containing them, which makes lookup by term fast. (Key point: Inverted index) 4. Which describes the build query and run query steps correctly? A. The query parser makes a structured query, the same analyzer is applied to the query text, then the searcher looks up the index and scores the matches B. The crawler fetches the query and the index writer ranks the results C. The query is run directly on the raw documents, then analyzed D. Query text skips the analyzer so that it matches exactly Answer: A. The query is parsed into a structured query using the same analysis as indexing, then matched against the inverted index and ranked (TF-IDF, BM25). (Key point: Query building and running) ## Related in other subjects - [Typical water supply system and components](https://www.bctengineer.com/question.md?key=semester_5%2Fwater_supply_engineering%3Atypical-ws-system-components) — Water Supply Engineering, chapter 1, asked 4× - [Typical power system network and components](https://www.bctengineer.com/question.md?key=semester_4%2Fpower_system_analysis_i%3Atypical-power-system-network-components) — Power System Analysis I, chapter 1 - [Searching: BFS vs DFS comparison](https://www.bctengineer.com/question.md?key=semester_6%2Fartificial_intelligence%3Ablind-search-dfs-bfs-compare) — Artificial Intelligence, chapter 3, asked 5× - [Passive components behavior and application protocols](https://www.bctengineer.com/question.md?key=semester_7%2Frf_and_microwave_engineering%3Amicrowave-components-behavior-protocols) — RF and Microwave Engineering, chapter 1