- 2074 Bhadra · 4+4 marks
MapReduce vs Hadoop; IR process
Short answer
MapReduce is a programming model; Hadoop is a framework including HDFS and MapReduce. IR process: collect, preprocess, build inverted index, match query, rank results. For 'cloud storage', D2 'cloud storage is cheap and safe' ranks higher than D1 'cloud computing is cheap'.
Differentiate between Map reduce and Hadoop systems. Explain the process of Information Retrieval in detail with example.
Difference between MapReduce and Hadoop
| Basis | MapReduce | Hadoop |
|---|---|---|
| Nature | A programming model and processing technique | A complete open-source framework or ecosystem |
| Origin | Introduced by Google (2004) | Apache project, inspired by Google's MapReduce and GFS |
| Purpose | Process large data in parallel using map and reduce functions | Store (HDFS) and process (MapReduce, YARN) big data on clusters |
| Components | Map, shuffle/sort, reduce | HDFS, YARN, MapReduce, plus tools such as Hive, Pig, HBase |
| Storage | Does not store data itself | HDFS stores data |
| Relation | Is a component of Hadoop | Implements MapReduce |
Process of information retrieval
Information retrieval (IR) finds documents in a collection that are relevant to a user's query.
- Document collection and acquisition: crawl or collect documents.
- Preprocessing: tokenisation, stop word removal, stemming, normalisation.
- Indexing: build an inverted index (term -> documents with frequency/positions).
- Query formulation: the user states a need; the query is processed in the same way.
- Matching and retrieval: the engine finds documents containing the terms.
- Ranking: documents are scored (TF-IDF, BM25, PageRank) and sorted.
- Presentation and feedback: show results; relevance feedback may refine the query.
Docs -> Preprocess -> Index
Query -> Preprocess -> Match -> Rank -> Results
Example
Documents: D1 "cloud computing is cheap", D2 "cloud storage is cheap and safe". Inverted index: cloud -> D1, D2; computing -> D1; storage -> D2; cheap -> D1, D2; safe -> D2. Query "cloud storage" matches D2 on two terms and D1 on one, so D2 is ranked first, then D1.
✓ Answer checked by Claude Opus 5.5 (high effort)
Related questions
- MapReduce vs Hadoop · Information System, chapter 8
- Information retrieval in cloud · Information System, chapter 7
- Distributed system and Hadoop · Information System, chapter 8
- MIS vs DSS information and processing · Information System, chapter 4
- MapReduce working with example · Information System, chapter 8
- MapReduce with GFS and HDFS · Information System, chapter 8
- Web documents, browser, information retrieval · Web Technology, chapter 2
- Differentiate wired and wireless systems · Basic Electronics Engineering, chapter 5 · asked 2×
- MapReduce worker failure tolerance · Big Data Technologies, chapter 3
- MapReduce workflow with example · Big Data Technologies, chapter 3 · asked 2×