Why Naive RAG Fails for Codebases

Most "chat with your repo" products are thin wrappers around the same pipeline: chunk text, embed chunks, retrieve top‑k neighbors, stuff them into a prompt, pray.

That architecture is fine for policy PDFs. It is actively harmful for codebases.

The failure mode is structural, not parametric

Naive RAG treats source code like prose. A 400‑line React component becomes four overlapping 512‑token windows. A Stripe webhook handler sits in the same embedding neighborhood as a unrelated fetch() wrapper because both mention "session", "user", and "POST".

The model does not retrieve the wrong answer because your top_k is 5 instead of 8. It retrieves the wrong answer because the unit of retrieval is wrong.

// Two functions. Similar English if you embed docstrings + bodies naively.
export async function createCheckoutSession(userId: string) {
  return stripe.checkout.sessions.create({
    mode: "subscription",
    client_reference_id: userId,
  });
}
 
export async function destroyUserSession(userId: string) {
  await db.sessions.delete({ where: { userId } });
}

Ask: "Where is checkout created?" A vector index built on sliding windows will happily surface destroyUserSession — it is semantically adjacent noise.

Code has grammar; paragraphs do not

Identifiers carry precision. Boundaries matter. A naive chunker:

Splits mid‑function, leaving half a closure in one chunk and its error handler in another.
Merges unrelated symbols when files are small enough to fit one window.
Ignores graph structure — importers, call sites, and interface implementations disappear.

AST‑aware chunking fixes the unit of work:

// Pseudocode: chunk by symbol, not by character count
for (const symbol of parseSymbols(file)) {
  const checksum = md5(symbol.sourceText);
  if (cache.has(checksum)) reuseExplanation(checksum);
  else explain(symbol);
}

When retrieval keys off symbols (functions, classes, hooks) instead of arbitrary text spans, you get stable cache keys, incremental updates, and explanations that map to line ranges — not approximate file offsets.

Embeddings confuse topic with responsibility

Embeddings encode distributional similarity. Code retrieval needs causal and structural similarity:

Question	What naive RAG returns	What you need
"Who calls `validateJwt`?"	Chunks mentioning JWT	Call graph edges
"Where is Pro pricing defined?"	Marketing copy + env vars	Symbol + constant refs
"What breaks if I change this DTO?"	Random TypeScript interfaces	Type references + importers

Without a call graph (even a heuristic one), RAG becomes expensive fuzzy grep — with hallucination on top.

Prompt stuffing scales cost, not accuracy

The naive fix is bigger context windows: retrieve 40 chunks instead of 4. You pay linearly in tokens while signal-to-noise ratio collapses. The LLM sees ten partial implementations of "auth" and synthesizes a fifth, imaginary one.

Production-grade code intelligence inverts the loop:

Parse → symbols with checksums
Retrieve → graph‑aware neighbors + exact symbol hits
Explain → one minified JSON payload per changed symbol
Ground → every sentence links to path:line-range

That is how you keep API spend bounded and stop lying to senior engineers.

What actually works

We built AnnoTrace around constraints naive RAG ignores:

Content-addressable AST chunks — MD5 per symbol; skip unchanged code entirely.
Batched, schema-minified LLM calls — explain only cache misses.
Bidirectional traceability — English layer ↔ exact source ranges.
Incremental re-analysis — edit one hook, re-explain one hook.

If your toolchain cannot tell you which symbol it retrieved and which lines that symbol spans, it is not a code understanding system. It is a summarizer with a vector index costume.

Next: wire AST-precision ingestion into CI so every push updates explanations for changed symbols only — not your entire monorepo bill.