Idea #1
Pre-rendering document pages to screenshots during the indexing pipeline — and storing those screenshots alongside text chunks — is a meaningful quality lever for enterprise RAG on complex documents, and it is often skipped. Schematics, multi-column PDFs, financial tables, and scanned images are not built for machine text extraction: OCR tools like Textract or Tesseract frequently corrupt table structure, merge columns, and strip spatial layout information. When an LLM reads that degraded text, it inherits the layout errors — and spatial reasoning failures compound downstream (a misaligned table row becomes a wrong answer). Generating screenshots at parse time costs little and provides a fallback: the agent can visually inspect a page to verify whether a text chunk's spatial context is correct, or to retrieve data from a chart that has no clean text representation. In the demo, each chunk in the retrieval results was paired with its corresponding page screenshot, giving the agent dual-modality evidence to cross-check before committing to an answer.
Idea #2
Hybrid retrieval — BM25 keyword search fused with vector/semantic search, followed by a reranking pass — consistently outperforms either method alone and should be the default for any enterprise search system. Keyword search is precise for exact terms (product codes, legal clause numbers, financial line items) but blind to paraphrase and semantic intent. Vector search captures meaning and handles synonym variation but can miss exact-match precision, especially for numeric or domain-specific strings. Fusing the two with weighted scoring and then reranking the top-N results using an LLM or cross-encoder model eliminates the blind spots of each. In benchmarking data cited, the fusion approach beats raw keyword or raw vector baselines, and the reranking step compounds the gain. The practical implication: a single-modality retrieval system (even a good vector store) is leaving accuracy on the table for enterprise document search, and the implementation cost of adding BM25 and a reranker is low relative to the quality gain.
Idea #3
Data freshness and permission metadata synchronization are the two operational blockers that cause pre-indexed RAG systems to fail in production, and they need to be treated as first-class engineering concerns rather than afterthoughts. Freshness is hard: every document update in a live corpus may require re-parsing (especially if the document format changed), re-chunking, and re-embedding — and if the pipeline is slow or expensive, indexes drift out of sync with the source. This is cited as a key reason Anthropic and other companies have hesitated to adopt pre-indexing for their own internal tools. Permission metadata compounds this: enterprise corpora in Google Drive or SharePoint carry per-file ACLs, and if those aren't captured and stored in the vector index at ingestion time, every query either ignores permissions (a security risk) or triggers a live permission lookup (a latency and scalability problem). The recommended pattern is a three-stage sync pipeline — ingestion, parsing to a canonical format (markdown), chunking/indexing — with permission metadata pushed into the vector store as filterable fields during ingestion so queries can filter on access rights without re-indexing. Turbopuffer is the vector store cited as supporting this pattern at the multi-tenant, tens-of-millions-of-documents scale.
Idea #4
Whether to use local file traversal (grep/cat) or pre-indexed vector search is a scale and data-structure decision, not a philosophical one — and getting it wrong in either direction is expensive. Local traversal works when a corpus is small (roughly 100–1,000 files), semi-structured, and already organized with implicit hierarchy (like a code repo where imports and test files leave breadcrumbs). Pre-indexing becomes necessary when you cross into thousands or millions of files, when files are heterogeneous formats (PDFs, PowerPoints, scanned schematics mixed together), and when the corpus has no meaningful folder hierarchy an agent can navigate. The cost of the wrong choice: local traversal at enterprise scale burns tokens on every run re-reading files that haven't changed; pre-indexing at code scale introduces a sync problem (keeping the index current as code changes) with no payoff. Claude Code's decision to skip vector search entirely is a case study in the former: code repos are small, syntactically structured, and already on disk — the three conditions that make pre-indexing overhead not worth it. Switch to insurance, finance, or healthcare document lakes and all three conditions break down.
Idea #5
The right agentic retrieval architecture is not 'RAG or file traversal' — it is both, layered in a specific order: semantic/hybrid search as a compass to identify which document subset is relevant, then tool-based local traversal (list files, grep with offset, read specific ranges) to extract precise grounded answers. Semantic search alone leads to hallucination when exact figures matter, because chunk boundaries often cut across the data the agent needs. File traversal alone burns tokens at scale and breaks when documents aren't downloaded locally. The layered approach lets the agent narrow from millions of files to a handful using embeddings, then operate precisely within that subset without needing to stream the full documents to its context. In the demo using 135 Alphabet financial report files (HTM and PDF mixed), hybrid search returned relevant chunks plus page screenshots; the agent then issued grep commands with explicit character offsets and lengths to pull exact table rows — giving it grounded, verifiable answers rather than reconstructed ones from chunk approximations.
Source: Grep or Embeddings? Agentic Search Over Company Documents — George He, LlamaIndex (AI Engineer)
Text summarized and optimized using Anthropic’s models and reviewed by a human.