Case Study
Newsroom Discovery
Rebuilt a decade-old open-source research technique into a working, self-hosted product for investigative journalists — one that finds documents about a topic using AI similarity search, including the ones a keyword search would completely miss.
Client
Independent / Self-directed
Year
2026
Domain
Problem
Investigative reporting on large document leaks depends on keyword search, which only surfaces documents that share the reporter's exact vocabulary. A well-known open-source technique (doc2vec, applied by Quartz to the Mauritius Leaks) solves this by finding documents that are topically similar even without shared keywords — but the reference implementation was a research notebook: hand-edited CSVs, curl commands, and cell-by-cell execution, with years of dependency rot (Python 3.14 incompatibilities, a discontinued Elasticsearch Homebrew formula, deprecated library APIs) standing between it and actually running. None of that was usable by an actual reporter.
Approach
Revived the reference notebook
Got a years-old, broken research notebook running again on a real corpus (4,251 pages of NYC City Hall emails released under FOIL) — fixing Python 3.13+ incompatibilities in gensim, a discontinued Elasticsearch Homebrew formula, and deprecated PyPDF2/gensim APIs along the way.
Turned notebook cells into a product
Replaced hand-edited CSVs, curl commands, and cell-by-cell execution with a real Next.js + FastAPI application: type a keyword, select a few results, and click "find documents like these" — with a visible badge showing which similar results do and don't contain the literal search term.
Replaced brute-force search with a vector database
Swapped the original in-memory, exact-comparison similarity search for Qdrant running embedded (no server, no new infrastructure) — with a backfill migration so the original validated results kept working, and incremental embedding so newly-added documents are searchable without retraining the model.
Self-hosted ingestion and OCR
Added Apache Tika and Tesseract for extracting and OCR'ing Word docs, spreadsheets, emails, and scanned images — entirely self-hosted, since the whole point is that sensitive leaked documents never touch a third-party API.
A safe way to demo it
Built a second, password-gated deployment on Vercel using real (not fabricated) fixture data, so the tool can be shared publicly without exposing the self-hosted backend or the real underlying corpus.
Outcomes
Self-hosted, end to end
Keyword search, similarity search, OCR, and the vector database all run locally — zero cloud API calls, zero telemetry.
4,251 → 3
Out of every page in the corpus, semantic similarity search narrowed the field to 3 genuinely relevant matches — each topically connected to a seed document, without sharing its exact keyword.
Notebook → product
Went from a broken, years-old research script to a working search tool, ingestion pipeline, and shareable public demo in one session.
Artifacts
Similarity search with keyword-miss badges
Selecting a few known-relevant pages surfaces genuinely similar documents, each flagged for whether it actually contains the search term — the whole point of the tool made visible.
Self-hosted ingestion + OCR
Upload a Word doc, spreadsheet, email, or scanned image; Apache Tika and Tesseract extract and OCR it locally, and it's searchable within seconds.
Curated leads → editor handoff
A reporter's shortlist of real leads, each with a note on why it matters, exported as a clean PDF + summary CSV — no raw scores or code required on the editor's end.