Lucene architecture and indexing approach

Apache Lucene is a high-performance, open-source full-text search library written in Java. It has no crawler and no user interface: it provides the core API to index and search text. Solr and Elasticsearch are built on it.

Architecture

 Data source -> Document -> IndexWriter -> Index (Directory)
                 (Fields)    + Analyzer         |
                                                v
 User query -> QueryParser -> Query -> IndexSearcher -> TopDocs
              + Analyzer                (IndexReader)   (hits)

Main classes:

• Document: the unit of indexing and search; a set of Fields (name/value pairs such as title, content, date). Fields can be stored, indexed and tokenized.
• Analyzer: converts field text into tokens (a tokenizer plus filters such as lower-casing, stop-word removal and stemming).
• IndexWriter: creates and updates the index; it adds, deletes and updates documents.
• Directory: where the index lives (FSDirectory on disk, RAMDirectory in memory).
• IndexReader / IndexSearcher: open the index and run searches.
• QueryParser / Query: parses the user's query string into Query objects (TermQuery, BooleanQuery, PhraseQuery, WildcardQuery, RangeQuery).
• TopDocs / ScoreDoc: the ranked results with their scores.

Indexing approach: the inverted index

Lucene uses an inverted index: a mapping from each term to the documents that contain it (like the index at the back of a book), instead of a map from each document to its words.

Steps:

1. Create a Document with fields from the source record.
2. The Analyzer tokenizes the text, lower-cases it, removes stop words and stems the words.
3. IndexWriter builds the term dictionary and the postings lists (doc id, term frequency, positions) and writes them to the index.

Example: Doc1 = "Big data tools", Doc2 = "Data is big", Doc3 = "Hadoop tools".

A query "big AND tools" intersects the postings lists of big and tools and returns Doc1.

Segments

The index is made of immutable segments. New documents are buffered in memory and flushed to disk as a new small segment. Background merging combines segments into larger ones and removes deleted documents. This makes writing fast; a search reads all the segments.

Scoring

Each hit is scored by relevance using TF-IDF (in older versions) or BM25 (the current default): terms that are frequent in the document but rare in the collection give a higher score.