- 2073 Magh · 5 marks
Short note: Lucene analyzers
Short answer
Lucene analyzers turn field text into tokens using a tokenizer followed by optional filters like lowercasing and stemming. Common analyzers include StandardAnalyzer (word boundaries, lowercasing), SimpleAnalyzer (non-letters split), WhitespaceAnalyzer, KeywordAnalyzer (single token), and language-specific ones like EnglishAnalyzer.
Write a short note on Analyzers available in Lucene.
An Analyzer in Lucene converts the text of a field into a stream of tokens (terms) that are added to the index. The same analyzer should be used when indexing and when parsing a query, so that the terms match. An analyzer consists of a Tokenizer (splits the text) followed by zero or more TokenFilters (modify the tokens), e.g. lower-casing, stop-word removal and stemming.
Common analyzers
| Analyzer | Behaviour |
|---|---|
| StandardAnalyzer | Splits on word boundaries (Unicode rules) and lower-cases. Older versions also removed common English stop words; from Lucene 8 its default stop-word set is empty. Default and general-purpose. |
| SimpleAnalyzer | Splits at non-letters and lower-cases. Digits and punctuation are dropped; stop words are not removed. |
| WhitespaceAnalyzer | Splits only on whitespace; keeps case and punctuation. |
| StopAnalyzer | Like SimpleAnalyzer, but also removes stop words. |
| KeywordAnalyzer | Treats the whole field as a single token; used for IDs and codes. |
| Language analyzers (EnglishAnalyzer, etc.) | Language-specific stop words and stemming. |
Example
Text: "The Quick Brown Foxes are Running"
- WhitespaceAnalyzer:
The, Quick, Brown, Foxes, are, Running - SimpleAnalyzer:
the, quick, brown, foxes, are, running - StandardAnalyzer:
the, quick, brown, foxes, are, running(Lucene 8+);quick, brown, foxes, runningin older versions, which removed stop words - EnglishAnalyzer:
quick, brown, fox, run(stop words removed and stemmed) - KeywordAnalyzer:
The Quick Brown Foxes are Running
Custom analyzers
A custom analyzer can combine any tokenizer and filters (e.g. a synonym filter, a stemmer, an n-gram filter) to suit the application. The choice of analyzer affects search quality: a good analyzer improves recall (it finds "run" for "running") without hurting precision.
✓ Answer checked by Claude Opus 5.5 (high effort)