The Content & Data Stack

Your model is a commodity; your context is not. Two startups can call the same frontier model with the same prompt and ship products that feel a generation apart, and the difference is almost never the prompt — it is what got retrieved and stuffed into the window before generation began. This chapter is about that machinery: how you turn a messy corpus of documents into vectors, where you keep them, how you search them well, and how you stop the whole thing from silently rotting the day after you launch.

We assume you already know what RAG is and how the retrieved context lands in a prompt (see Chapter 3), which embedding provider you lean toward (see Chapter 7), and that the infrastructure to run all this is Chapter 9’s problem. Here we go deep on the plumbing itself, and we do it through Cortex, our running example: an autonomous customer-support agent for mid-market SaaS companies. Cortex ingests each customer’s help center, product docs, and historical support tickets, then answers live conversations. If its retrieval is wrong, it hallucinates a refund policy that does not exist — so for Cortex, the content stack is not a supporting system. It is the product.

8.1Embeddings and chunking done right#

An embedding turns a span of text into a vector so that semantically similar spans land near each other. The chunk — the unit of text you embed and later retrieve — is the single most consequential design decision in the stack, and the one teams most often get wrong. One systematic 2026 analysis attributed roughly 80% of RAG failures to the ingestion and chunking layer rather than the LLM.¹ When retrieval fails, you are not debugging the model; you are debugging how you cut up your documents nine steps upstream.

The naive move — split every document into fixed 1,000-token blocks — destroys meaning at the boundaries and buries the signal. The 2026 production default is narrower and smarter: start with a recursive splitter at 400–512 tokens with 10–20% overlap, respecting natural boundaries (headings, paragraphs, list items), and escalate to semantic or page-level chunking only when your retrieval evals (see Chapter 10) prove you need it.² Two nuances matter. First, overlap is not free: a January 2026 study found it often provided no measurable benefit while inflating index size and cost, so treat it as a knob to test, not a default to trust.² Second, there is a context cliff around 2,500 tokens where answer quality degrades as chunks get bigger — bigger is not safer.²

The pattern that resolves the tension is hierarchical (or parent-document) chunking: embed small child chunks for precise matching, but return the larger parent section to the model for context. You find with a scalpel and feed with a ladle. This is now the most widely adopted production pattern.² For Cortex, a child chunk is a single paragraph of a help article; the parent is the whole article section, so a hit on “how do I rotate my API key” retrieves the exact sentence but hands the model the full key-management section.

The highest-leverage upgrade to embeddings is contextual retrieval: before embedding each chunk, prepend a short, LLM-generated sentence situating it in its source document (“This chunk is from the Billing article, section Refunds, and describes the 30-day window”). Anthropic reported that contextual embeddings alone cut retrieval failures ~35%; pairing them with contextual BM25 (hybrid search, covered below) reaches ~49%; adding reranking reaches ~67%.³ It costs one cheap LLM call per chunk at ingest time — trivially amortized against every future query.

On the embedding model itself — a Chapter 7 decision — two 2026 facts shape your chunking economics. Matryoshka representation lets one model emit 2048/1024/512/256-dimensional vectors from a single pass, so you can truncate to smaller dimensions and shrink storage without re-embedding.⁴ And quantization is dramatic: voyage-3-large at int8 precision and 512 dimensions scored ~8.6% higher than OpenAI’s full 3,072-dim float32 vectors while using roughly one-twenty-fourth the storage.⁴ Binary quantization cuts storage a further 32× versus float32 at equal dimensions.⁵ The lesson: default to 768–1,024 dimensions, reach for quantization early, and never pay to store 3,072-dim float32 vectors unless an eval says you must.

8.2Choosing a vector database or store#

The market has fragmented into good options, so the wrong-answer surface has shrunk — but the “defensible default” question is sharper than ever. Do not start by asking which vector database is best. Ask how many vectors you will have, whether you already run Postgres, and whether you need to filter retrieval by metadata (tenant, product, recency) — for a multi-tenant product like Cortex, that last one is non-negotiable.

For most workloads under a few million vectors, pgvector inside your existing Postgres is the strongest choice: embeddings, source documents, and metadata live in one database you query with ordinary SQL joins, and with the pgvectorscale extension it stays competitive with dedicated engines up to roughly 50–100M vectors.⁶ You avoid a second datastore, a second consistency model, and a second thing to back up. The moment you outgrow it is a good problem, and the migration is mechanical.

The load-bearing column is hybrid search — Weaviate, Qdrant, Turbopuffer, and Elasticsearch support it natively.⁷ If your product hinges on retrieval quality (it does), pick a store whose hybrid support is native rather than bolted on, or accept that you will run a separate keyword index alongside your vectors. Cortex runs pgvector per-tenant in early stage — each customer’s corpus is small, and Postgres row-level security gives clean tenant isolation for free — with a documented path to Turbopuffer once any single tenant crosses tens of millions of vectors.

8.3RAG architecture patterns in production#

A demo RAG pipeline is embed(query) → top-k → prompt. A production one is a graph with an ingestion side and a serving side that share nothing but the index, and the diagram below is the spine every serious system converges on.

Three patterns separate production from prototype. First, retrieve more than you send. Pull 50–100 candidates from hybrid retrieval, then let a reranker (below) cut to the 5–10 that actually enter the prompt. Cheap recall up front, expensive precision at the end. Second, always return citations. Every generated claim should carry the chunk ID it came from, both so users can verify and so you can trace failures back to a specific document. For Cortex, an answer without a citation to a help article is a bug, not a stylistic choice — it is how a support agent earns trust instead of confidently inventing policy. Third, filter before you search, not after. Push tenant, product-version, and recency filters into the vector query so you never rank a chunk you would have to discard — critical when tenant B must never see tenant A’s tickets.

Beyond baseline retrieval, two patterns are worth knowing but not defaulting to. Query rewriting — having a cheap model rephrase or decompose the user’s question before retrieval — reliably helps on conversational and multi-part queries; Cortex uses it to turn “it’s still broken” into a self-contained search query using conversation history. Agentic / corrective RAG, where the system retrieves, judges the result, and re-retrieves if confidence is low, buys accuracy at the cost of latency and token spend; reserve it for high-stakes answers, not every “where’s my invoice.” Benchmarks on text-and-table corpora confirm the two-stage hybrid-plus-reranker pipeline as the recommended baseline, with corrective loops as an upgrade, not a starting point.⁹

8.4Data pipelines, freshness and sync#

The demo indexes a corpus once. The product indexes a corpus that changes — help articles get edited, policies get rewritten, tickets get resolved — and here is the failure mode nobody sees coming: semantic similarity has no temporal dimension. A stale embedding scores exactly as high as a fresh one, so an out-of-date chunk gets retrieved with full confidence and the model answers from last quarter’s refund policy. There is no error, no exception, no red line on a dashboard. Retrieval quality decays silently, and teams solve freshness last, usually after a customer catches a wrong answer.¹⁰

The batch answer — re-embed the whole corpus nightly — is wasteful and always a few hours stale. The 2026 production answer is event-driven incremental sync via change data capture (CDC): tools like Debezium or Striim mine the database transaction log, and every insert, update, and delete emits a change event that flows through a streaming layer to re-embed only what changed, reflecting edits in the index within seconds.¹⁰ You process deltas, not the corpus.

The operation everyone forgets is deletes. When a help article is retired, its chunks must leave the index immediately, or Cortex will keep citing a page that 404s. Your pipeline needs an explicit tombstone path — a delete event removes every chunk whose source_id matches — and updates must be handled as delete-then-insert so old chunk versions do not linger as ghosts alongside new ones. This is why stable, deterministic chunk IDs derived from source identity are not optional.

A practical ingest contract for Cortex, expressed as the record every chunk carries:

# Every chunk is idempotent, source-linked, and versioned.
chunk = {
    "id": stable_hash(source_id, chunk_index, content_version),
    "text": contextualized_text,      # child chunk + prepended context
    "embedding": embed(contextualized_text, dim=1024, dtype="int8"),
    "metadata": {
        "tenant_id": "acme",          # filter before search — hard isolation
        "source_id": "help/refunds",  # enables delete + reconcile
        "source_uri": "https://...",  # citation target
        "version": content_version,   # supersede old chunks on update
        "updated_at": "2026-07-30",   # temporal signal against index rot
    },
}
# On a CDC update event: delete WHERE source_id, then upsert new chunks.

Run a periodic reconciliation job regardless of how good your event stream is: walk the source of truth, confirm every live document has current chunks and every retired one has none. Streams drop messages; reconciliation is the backstop that turns silent rot into a caught discrepancy. Budget for it before you launch, not after the first embarrassing answer.

8.5Reranking, hybrid search and retrieval quality#

Pure vector search has a blind spot that surprises people: it does not know words. Dense embeddings capture meaning, so they nail “automobile repair” ≈ “car maintenance” — but they fumble the exact tokens that matter most in support: product SKUs, error codes, API method names, version strings, a specific customer’s account ID. On these, the decades-old keyword algorithm BM25 beats dense retrieval, and 2026 benchmarks confirm BM25 outperforming even strong commercial embedding models on most metrics when queries contain rare, distinctive terms.⁷ For a Cortex ticket that says “getting ERR_QUOTA_402 on the v3 batch endpoint,” you want the chunk with that exact error code, and keyword search finds it while pure vectors drift toward “quota-related billing issues” in general.

Hybrid search runs both — dense and BM25 — and fuses the rankings, typically with reciprocal rank fusion (RRF), which combines result lists without needing to normalize incompatible score scales. You get semantic recall and exact-match precision from the same query. This is the recall stage.

The precision stage is a reranker: a cross-encoder that scores each candidate against the query jointly (reading query and document together, not as two independent vectors), which is far more accurate but too slow to run over a whole corpus — so you run it over only the ~50–100 candidates hybrid search surfaced. The 2026 numbers make the case unambiguous. Hybrid retrieval plus a Cohere reranker delivered a +17.4% relative Recall@5 over hybrid RRF alone and +39% over dense-only on a text-and-table benchmark.⁹ On a financial dataset, Cohere Rerank 3.5 scored +23.4% over hybrid search and +30.8% over BM25 alone.⁸ The tax is latency: reranking adds roughly 100–300ms per query (a hybrid-plus-Cohere top-50 pipeline measured p50 ≈ 230ms, p95 ≈ 410ms).⁷

For Cortex the full serving path is: rewrite the query using conversation context → hybrid retrieve 50 candidates (dense + BM25, RRF-fused, tenant-filtered) → rerank to the top 8 → generate with citations. The reranker is where quality is won: it is the difference between surfacing the article that mentions refunds and the one that states the 30-day window. Whether that pipeline actually improves answers — versus just feeling more sophisticated — is a question only retrieval evals can settle, and that is Chapter 10’s domain. Build the pipeline here; prove it there.