Keyword Search (BM25)

Keyword Search (BM25) Definition

BM25 (Best Matching 25) is a classic, probabilistic ranking algorithm used by search engines (like Elasticsearch and OpenSearch) to estimate the relevance of a document to a given search query based on literal word occurrences.

How It Works: BM25 scores a document based on three main factors

• Term Frequency (TF): How often a search term appears in the document.
• Inverse Document Frequency (IDF): How rare the term is across the entire database (e.g., "the" is ignored, while "quantum" is heavily weighted).
• Document Length Normalization: Shorter documents are favored over longer ones to prevent keyword stuffing.

Strengths

• Fast and Cheap: Extremely fast to index and query; requires minimal CPU and memory.
• Exact Matches: Highly accurate for specific identifiers, serial numbers, error codes, and unique brand names (e.g., searching "iPhone 15 Pro Max" or "ERR_404").

Weaknesses

• Vocabulary Mismatch: It cannot understand synonyms or intent. If you search for "sofa," a document containing only the word "couch" will not be retrieved.