---
title: "Lucene architecture and indexing approach"
subject: "Big Data Technologies"
chapter: 5
key: "semester_8/big_data_technologies:lucene-architecture-indexing"
source: https://www.bctengineer.com/q/semester_8/big_data_technologies/lucene-architecture-indexing
site: BCT Engineering
---

# Lucene architecture and indexing approach

Big Data Technologies · Chapter 5: Searching and Indexing Big Data

- Asked in: 2073 Bhadra, 10 marks
- Video: [Apache Lucene Open-Source Search Library Overview for beginners | KaaS-8](https://www.youtube.com/watch?v=TJ8ECPlz238) — Last Mile Developer
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_8/big_data_technologies/lucene-architecture-indexing
- Whole chapter: https://www.bctengineer.com/md/semester_8/big_data_technologies/chapter-5

## Question

Explain LUCENE architecture and its data indexing approach.

## Short answer

Lucene is a Java search library built around Document, Analyzer, IndexWriter, and IndexSearcher. Indexing creates an inverted index from tokenized text and stores immutable segments. Searching reads all segments and ranks documents using BM25 scoring.

## Answer

**Apache Lucene** is a high-performance, open-source full-text search library written in Java. It has no crawler and no user interface: it provides the core API to index and search text. Solr and Elasticsearch are built on it.

#### Architecture

```text
 Data source -> Document -> IndexWriter -> Index (Directory)
                 (Fields)    + Analyzer         |
                                                v
 User query -> QueryParser -> Query -> IndexSearcher -> TopDocs
              + Analyzer                (IndexReader)   (hits)
```

Main classes:

- **Document:** the unit of indexing and search; a set of **Fields** (name/value pairs such as title, content, date). Fields can be stored, indexed and tokenized.
- **Analyzer:** converts field text into tokens (a tokenizer plus filters such as lower-casing, stop-word removal and stemming).
- **IndexWriter:** creates and updates the index; it adds, deletes and updates documents.
- **Directory:** where the index lives (`FSDirectory` on disk, `RAMDirectory` in memory).
- **IndexReader / IndexSearcher:** open the index and run searches.
- **QueryParser / Query:** parses the user's query string into Query objects (TermQuery, BooleanQuery, PhraseQuery, WildcardQuery, RangeQuery).
- **TopDocs / ScoreDoc:** the ranked results with their scores.

#### Indexing approach: the inverted index

Lucene uses an **inverted index**: a mapping from each term to the documents that contain it (like the index at the back of a book), instead of a map from each document to its words.

Steps:

1. Create a `Document` with fields from the source record.
2. The `Analyzer` tokenizes the text, lower-cases it, removes stop words and stems the words.
3. `IndexWriter` builds the term dictionary and the **postings lists** (doc id, term frequency, positions) and writes them to the index.

Example: Doc1 = "Big data tools", Doc2 = "Data is big", Doc3 = "Hadoop tools".

| Term | Postings |
|---|---|
| big | Doc1, Doc2 |
| data | Doc1, Doc2 |
| hadoop | Doc3 |
| tools | Doc1, Doc3 |

A query "big AND tools" intersects the postings lists of `big` and `tools` and returns Doc1.

#### Segments

The index is made of immutable **segments**. New documents are buffered in memory and flushed to disk as a new small segment. Background **merging** combines segments into larger ones and removes deleted documents. This makes writing fast; a search reads all the segments.

#### Scoring

Each hit is scored by relevance using TF-IDF (in older versions) or BM25 (the current default): terms that are frequent in the document but rare in the collection give a higher score.

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. Which statement about Apache Lucene is correct?
   A. It is a web crawler with a built-in search UI
   B. It is a Java full-text search library that provides core index and search APIs, with Solr and Elasticsearch built on it
   C. It is a distributed file system
   D. It is a relational database
   Answer: B. Lucene is a library only: no crawling or UI, but the core API used by Solr and Elasticsearch. (Key point: What Lucene is)

2. In Lucene indexing, which pair of roles is correct?
   A. Analyzer writes the index to disk; IndexWriter creates tokens
   B. Analyzer converts field text to tokens; IndexWriter creates and updates the index
   C. IndexSearcher adds documents; IndexWriter runs queries
   D. QueryParser builds the term dictionary
   Answer: B. The Analyzer tokenizes, lower-cases, removes stop words and stems; IndexWriter builds postings and writes the index. (Key point: Role of IndexWriter and Analyzer)

3. With Doc1 = 'Big data tools', Doc2 = 'Data is big', Doc3 = 'Hadoop tools', what does the query 'big AND tools' return?
   A. Doc1 and Doc2
   B. Doc3 only
   C. Doc1 only
   D. Doc1, Doc2 and Doc3
   Answer: C. big is in Doc1, Doc2 and tools is in Doc1, Doc3; the intersection is Doc1. (Key point: Inverted index query)

4. Why does Lucene use immutable segments with background merging?
   A. New documents are flushed as small segments so writes are fast; merging combines segments and removes deleted documents
   B. Segments are edited in place so that deletes are instant
   C. Merging is needed because each segment can only hold one document
   D. Searches read only the newest segment
   Answer: A. Buffered documents are flushed as new segments, and merging keeps the number of segments low and drops deleted documents; searches read all segments. (Key point: Segments and merging)

## Related in Big Data Technologies

- [Short note: Lucene analyzers](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Ashort-note-lucene-analyzers) — Big Data Technologies, chapter 5
- [Components of indexing and searching](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Aindexing-searching-components) — Big Data Technologies, chapter 5, asked 2×
- [Google File System architecture](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Agfs-architecture) — Big Data Technologies, chapter 2, asked 2×
- [HBase architecture](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Ahbase-architecture) — Big Data Technologies, chapter 4, asked 2×

## Related in other subjects

- [Data warehouse architecture and components](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Adw-architecture-components) — Data Mining, chapter 1
- [National data warehouse process architecture](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Anepal-national-data-warehouse) — Data Mining, chapter 1
- [Architecture of data mining system](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Adm-system-architecture) — Data Mining, chapter 1
- [Short note: Likelihood approach to anomalies](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Alikelihood-anomaly-note) — Data Mining, chapter 6