How this search works

mehedi-search is an educational Java search engine. This demo searches a curated collection of English and Bengali reference summaries and student resource descriptions, bundled with the application. It does not call Google, Bing, or an AI model.

  1. At startup, load the saved corpus and tokenize each document: lowercase, preserve Unicode letters and numbers, and remove basic English stop words.
  2. Build an inverted index mapping tokens to documents and their term frequencies.
  3. For each query, apply the same tokenizer and score matching documents with BM25, using term frequency, rarity, and document length.
  4. Return up to ten results, sorted by score with document ID breaking ties. Scores are relevance weights, not confidence percentages.

The repository also implements TF-IDF, a bounded crawler with URL normalization and supplied robots rules, HTTP fetching, HTML extraction, and a MiniRedis-backed TTL search-result cache. These components have separate tests. The public HTTP demo wraps BM25 SearchService with the existing in-process MiniRedis cache (30-second TTL); it does not crawl on startup or on searches.

Limits and sources

This is a curated demo corpus, not full web search. Licensed summaries and locally authored resource descriptions are a saved snapshot, so coverage is limited and links or facts may change. Descriptions are not publisher quotations. Ranking tables, admission rates, tuition figures, and restricted datasets are not reproduced. Linked materials belong to their publishers. Matching is lexical: no stemming, synonyms, spelling correction, or semantic search. Multiple query terms use OR matching. The displayed result count is the number shown, not every match.

Source articles, revision links, and CC BY-SA attribution identify every indexed document. The repository separates the licensed source snapshot from metadata/link-only records and original descriptions. An offline build generates the corpus, source credits, and a dated machine-readable manifest. The index is rebuilt in memory on each restart.

Search · GitHub repository (access depends on repository permissions)