Retrieval Preparation
Our retrieval pipeline converts source information into structures that can be searched, filtered, ranked, and assembled into task-specific context. This occurs before a model is asked to reason over company history.
The preparation path
The preparation path can include document segmentation, chunking, metadata enrichment, embeddings, lexical indexes, semantic indexes, filters, reranking inputs, and retrieval evaluation.
Segmentation is workload-dependent
We do not treat chunk size as a universal constant.
A contract, email thread, financial report, call transcript, and operating procedure have different structural boundaries. The appropriate segmentation strategy depends on:
- Document structure
- Likely query type
- Semantic boundaries
- Context-window constraints
- Retrieval precision
- Recall
- Latency
Overly small chunks can lose the surrounding context required to interpret a statement. Overly large chunks can dilute retrieval precision and consume unnecessary model context. The correct unit is therefore determined by the workload, not by a default token count.
Index preparation
Different retrieval methods require different representations.
Semantic retrieval benefits from embedding representations that preserve conceptual similarity. Lexical retrieval requires an index that preserves exact terms. Structured filters may depend on metadata such as entity, document type, time, source, or business unit.
A prepared corpus can therefore maintain multiple access paths over the same underlying information.
- Semantic representation
- Lexical index
- Metadata
- Source / entity / time references
This supports hybrid retrieval without duplicating the meaning of the underlying record.
Retrieval evaluation
A retrieval system should be measured on whether it finds the evidence required for real tasks.
Improving embedding quality or changing chunking can alter both precision and recall. More retrieved text is not automatically better if it introduces irrelevant evidence into downstream context.
Retrieval preparation exists to make later reasoning selective rather than exhaustive.
The objective is not to turn every company document into vectors. It is to create a retrieval substrate capable of finding the right evidence, under the right constraints, for the workload being executed.