---
title: "Short note: Lucene analyzers"
subject: "Big Data Technologies"
chapter: 5
key: "semester_8/big_data_technologies:short-note-lucene-analyzers"
source: https://www.bctengineer.com/q/semester_8/big_data_technologies/short-note-lucene-analyzers
site: BCT Engineering
---

# Short note: Lucene analyzers

Big Data Technologies · Chapter 5: Searching and Indexing Big Data

- Asked in: 2073 Magh, 5 marks
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_8/big_data_technologies/short-note-lucene-analyzers
- Whole chapter: https://www.bctengineer.com/md/semester_8/big_data_technologies/chapter-5

## Question

Write a short note on Analyzers available in Lucene.

## Short answer

Lucene analyzers turn field text into tokens using a tokenizer followed by optional filters like lowercasing and stemming. Common analyzers include StandardAnalyzer (word boundaries, lowercasing), SimpleAnalyzer (non-letters split), WhitespaceAnalyzer, KeywordAnalyzer (single token), and language-specific ones like EnglishAnalyzer.

## Answer

An **Analyzer** in Lucene converts the text of a field into a stream of **tokens (terms)** that are added to the index. The same analyzer should be used when indexing and when parsing a query, so that the terms match. An analyzer consists of a **Tokenizer** (splits the text) followed by zero or more **TokenFilters** (modify the tokens), e.g. lower-casing, stop-word removal and stemming.

#### Common analyzers

| Analyzer | Behaviour |
|---|---|
| **StandardAnalyzer** | Splits on word boundaries (Unicode rules) and lower-cases. Older versions also removed common English stop words; from Lucene 8 its default stop-word set is empty. Default and general-purpose. |
| **SimpleAnalyzer** | Splits at non-letters and lower-cases. Digits and punctuation are dropped; stop words are not removed. |
| **WhitespaceAnalyzer** | Splits only on whitespace; keeps case and punctuation. |
| **StopAnalyzer** | Like SimpleAnalyzer, but also removes stop words. |
| **KeywordAnalyzer** | Treats the whole field as a single token; used for IDs and codes. |
| **Language analyzers** (EnglishAnalyzer, etc.) | Language-specific stop words and stemming. |

#### Example

Text: "The Quick Brown Foxes are Running"

- WhitespaceAnalyzer: `The, Quick, Brown, Foxes, are, Running`
- SimpleAnalyzer: `the, quick, brown, foxes, are, running`
- StandardAnalyzer: `the, quick, brown, foxes, are, running` (Lucene 8+); `quick, brown, foxes, running` in older versions, which removed stop words
- EnglishAnalyzer: `quick, brown, fox, run` (stop words removed and stemmed)
- KeywordAnalyzer: `The Quick Brown Foxes are Running`

#### Custom analyzers

A custom analyzer can combine any tokenizer and filters (e.g. a synonym filter, a stemmer, an n-gram filter) to suit the application. The choice of analyzer affects search quality: a good analyzer improves recall (it finds "run" for "running") without hurting precision.

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. What is an Lucene Analyzer made of?
   A. A Tokenizer followed by zero or more TokenFilters
   B. An IndexWriter and a Directory
   C. A QueryParser and a Searcher
   D. Only a stemmer
   Answer: A. A tokenizer splits text and token filters (lower-case, stop words, stemming) modify the tokens. (Key point: Analyzer structure)

2. Why should the same analyzer be used when indexing and when parsing a query?
   A. To make the index smaller
   B. So that terms in the query match the terms stored in the index
   C. Because Lucene refuses different analyzers
   D. To avoid using stop words
   Answer: B. If the analysis differs, query terms would not match indexed terms. (Key point: Same analyzer for index and query)

3. For a field holding IDs or codes that must stay intact as one token, which analyzer fits?
   A. SimpleAnalyzer
   B. WhitespaceAnalyzer
   C. EnglishAnalyzer
   D. KeywordAnalyzer
   Answer: D. KeywordAnalyzer treats the whole field value as a single token. (Key point: Choosing analyzers)