The problem

Pasting pages into chat breaks on context limits, loses structure, can’t follow links, and forgets everything between sessions. Retrieval for research needs to be persistent, recursive, and citation-faithful — and recursive crawling needs to be provably unable to run away on API credits.

Technical approach

A five-layer pipeline (ingestion → indexing → retrieval → orchestration → MCP server) behind three tools: answer, search, status. Semantic parent/child chunking, HNSW vector search with provider-agnostic reranking, LLM query decomposition with MMR dedup. Deep mode recurses through links-of-links under an 11-rule evaluator that halts on saturation or diminishing returns, capped by hard backstops. A fully free/local profile (Ollama embeddings, rule-based decomposition, raw fetcher) runs end-to-end with zero paid keys.

Technical highlights

• Guaranteed-citation design: verbatim quotes are reserved in the token budget and never silently dropped.
• Deep runs report their own traversal — depth reached, stop reason, and indexed-vs-reachable coverage.
• Both fetch paths respect robots.txt with per-origin rate limits.

Why it matters here

For ML/AI and infra audiences: retrieval infrastructure with the operational edges — cost ceilings, coverage reporting, citation fidelity — treated as first-class.