Question 1

What is Retrieval-Augmented Generation (RAG), and why is it important?

Accepted Answer

RAG is an architecture that gives an LLM access to external knowledge at inference time: for each query you first retrieve relevant documents from a knowledge source, then feed them into the prompt so the model generates an answer grounded in that retrieved context — instead of relying only on its frozen parametric memory.

Why it's important:
- Reduces hallucination: the model answers from provided facts, not guesses.
- Fresh & updatable knowledge: update the index, not the weights — beats the training cutoff.
- Domain/private data: ground answers in your proprietary corpus without training on it.
- Attribution: you can cite exactly which source a claim came from.
- Cost: far cheaper than fine-tuning for injecting knowledge.

ONE-LINER:
"RAG turns a closed-book exam into an open-book exam. The model stops relying on memory and starts reasoning over retrieved evidence — that's how you get current, private, and citable answers without retraining."

Reference in assignment: "AI Engineering Explained" video (LLM/RAG/MCP/Agent).

Question 2

Explain the architecture of a basic RAG system.

Accepted Answer

A basic RAG system has two phases: an offline indexing phase that prepares the knowledge base, and an online query phase that retrieves and generates per request.

Indexing (offline): Documents → Chunk → Embed → Store in vector DB
Query (online): User query → Embed → Similarity search → Top-k chunks → Assemble prompt (query + context) → LLM → Answer

Walkthrough:
1. Indexing: load documents, split into chunks, embed each chunk into a vector, store vectors + text + metadata in a vector database.
2. Retrieval: embed the user's query with the same model, find the nearest chunks by vector similarity (often + keyword/metadata filters).
3. Augmentation: insert the retrieved chunks into the prompt as context.
4. Generation: the LLM answers grounded in that context, ideally with citations.

DEPTH SIGNAL:
Mention that production RAG adds stages between these: query transformation before retrieval, and reranking + context assembly after retrieval. "Naive RAG" is retrieve-then-read; "advanced RAG" optimizes pre- and post-retrieval.

Question 3

What are the key components of a RAG pipeline?

Accepted Answer

Five core components: (1) a document/ingestion + chunking layer, (2) an embedding model, (3) a vector store / retriever, (4) the generator LLM, and (5) the orchestration that ties query → retrieval → prompt → answer together. Advanced pipelines add reranking, query transformation, and evaluation.

Components:
- Ingestion + chunking: Parse docs, split into retrievable units.
- Embedding model: Text → vectors (indexing & queries).
- Vector store / retriever: ANN similarity search, metadata filtering.
- Reranker (optional): Cross-encoder re-scores top candidates.
- Generator (LLM): Produces grounded, cited answer.
- Orchestration + eval: Pipeline glue, monitoring, RAGAS metrics.

FRAMING:
"Retriever quality caps the whole system — the generator can only be as good as the context it's handed. That's why most RAG improvement work is on retrieval (chunking, hybrid, rerank), not the LLM."

Question 4

Compare RAG vs fine-tuning. When would you use each?

Accepted Answer

RAG injects knowledge at inference time via retrieval; fine-tuning bakes behavior/style/skills into the weights via training. RAG is for "what the model knows"; fine-tuning is for "how the model behaves." They're complementary, not competitors.

Comparison:
- Best for: Facts, fresh/private knowledge, citations (RAG) vs Style, format, tone, narrow skills, latency (Fine-tuning).
- Update knowledge: Update the index, instant (RAG) vs Retrain, slow and costly (Fine-tuning).
- Hallucination: Lower, grounded (RAG) vs Not inherently reduced (Fine-tuning).
- Attribution: Yes, sources (RAG) vs No (Fine-tuning).
- Cost/infra: Vector DB + retrieval at query time (RAG) vs Training + longer prompts avoided (Fine-tuning).

Decision rule:
Use RAG when knowledge is large, changing, private, or needs citations. Use fine-tuning when you need consistent format/tone, a specialized skill, or to shrink prompts/latency. Use both when you want a domain-adapted model that also grounds on live data.

SAY THIS:
"Knowledge → RAG. Behavior → fine-tune. If someone reaches for fine-tuning to add facts, that's usually the wrong tool — RAG is cheaper, updatable, and citable."

Question 5

What are chunking strategies, and how do you choose the right chunk size?

Accepted Answer

Chunking is splitting documents into retrievable units. Chunk size is a precision/recall trade-off: small chunks retrieve precisely but may lack context; large chunks carry more context but dilute the embedding and retrieve noisily. The "right" size depends on your content and query type — there's no universal answer.

How to choose:
- Start with a sensible default: ~200–500 tokens with 10–20% overlap for prose.
- Match the query granularity: pinpoint fact lookups → smaller chunks; conceptual/summary queries → larger.
- Respect structure: split on natural boundaries (paragraphs, sections, headings), not mid-sentence.
- Decouple retrieval from generation: retrieve on small chunks, feed larger context (parent-child, Q7).
- Tune empirically: measure context precision/recall on an eval set and adjust.

WHY IT'S THE #1 LEVER:
"Chunking decides what the system is physically capable of retrieving. Everything downstream — embeddings, reranking, HyDE — works with whatever the chunks preserved. Bad chunks = polishing a compromised input. Fix chunking first."

Question 6

Compare fixed-size chunking, semantic chunking, and recursive chunking.

Accepted Answer

They differ in where they cut. Fixed-size splits every N tokens (with overlap) — simple, fast, but cuts mid-thought. Recursive splits hierarchically on a priority of separators (paragraph → sentence → word) to respect structure — the practical default. Semantic uses embedding similarity between sentences to cut at topic shifts — most coherent, but slow and costly.

Comparison:
- Fixed-size: How it cuts - Every N tokens + overlap. Pros/cons - Fast, simple; ignores meaning.
- Recursive: How it cuts - Hierarchical separators. Pros/cons - Best balance; LangChain default.
- Semantic: How it cuts - Embedding-similarity boundaries. Pros/cons - Most coherent; expensive, mixed ROI.

HONEST, SENIOR TAKE:
Multiple 2025–26 benchmarks found semantic chunking often doesn't justify its 10× cost — recursive/fixed frequently match it. So default to recursive, and only invest in semantic if your eval shows a real gain. Saying this signals you evaluate rather than cargo-cult.

Question 7

What is parent-child chunking, and how does it improve retrieval?

Accepted Answer

Parent-child (a.k.a. small-to-large / parent-document retrieval) decouples the retrieval unit from the generation unit: you embed and search over small "child" chunks for precision, but when a child matches, you return its larger "parent" chunk (the surrounding section/paragraph) to the LLM for context.

Index small children → search finds precise child → return the parent block to the generator

This solves the core tension: small chunks retrieve accurately but lack context; large chunks have context but retrieve fuzzily. Parent-child gets both — pinpoint retrieval and rich generation context. On long-form docs it often lifts faithfulness by several points on its own.

WHEN TO USE:
Great for long, structured documents (manuals, papers, policies). Less useful for short self-contained docs (FAQs, tweets) where there's no meaningful "parent." (Directly applicable to ThinkStudio/CareerLift-style corpora.)

Question 8

What are embedding models, and how do they convert text to vectors?

Accepted Answer

An embedding model is a neural network (usually a Transformer encoder) that maps a piece of text to a fixed-length dense vector capturing its meaning, such that semantically similar texts land close together in the vector space (measured by cosine similarity). It's the bridge that turns "meaning" into "geometry" so retrieval becomes nearest-neighbor search.

How:
1. Tokenize and run the text through the encoder to get contextual token vectors.
2. Pool them (mean-pool or [CLS]) into one vector.
3. Models are trained with contrastive learning: pull query–relevant-doc pairs together, push irrelevant ones apart.

At query time you embed the query the same way and retrieve chunks whose vectors are closest.

CRITICAL RULE:
Use the same embedding model for indexing and querying — mismatched models put text in incomparable spaces and retrieval silently breaks. (This is a classic RAG bug.)

Question 9

How do you choose an embedding model for your RAG system?

Accepted Answer

Balance retrieval quality, dimensionality/cost, context length, domain fit, and deployment constraints. Start from a benchmark like MTEB for retrieval tasks, then validate on your own data — leaderboard rank rarely transfers perfectly to your domain.

Selection checklist:
- Quality on your task: test top candidates on a labeled retrieval eval from your corpus.
- Domain match: specialized domains (legal, medical, code) may need domain-tuned models.
- Dimensionality: higher dims = better quality but more storage/latency; some models support Matryoshka truncation.
- Max sequence length: must cover your chunk size.
- Multilingual needs? Pick a multilingual model.
- Open vs API: privacy, cost-at-scale, self-hosting.

PROCESS OVER PICKING:
"I shortlist from MTEB, then run a head-to-head on my own queries measuring recall@k and context precision. The best leaderboard model isn't always best on your data — always validate in-domain."

Question 10

What is the role of metadata filtering in RAG systems?

Accepted Answer

Metadata filtering constrains vector search to a subset of the index using structured attributes attached to each chunk (source, date, author, department, doc type, language, access level). It combines semantic similarity with hard filters — "find chunks similar to X where date > 2024 and dept = finance."

Why it's essential:
- Precision: narrows the search space, cutting irrelevant matches.
- Freshness: filter to recent docs to avoid stale answers.
- Access control: restrict to documents the user is allowed to see (see Q30).
- Routing: scope to a specific product/tenant/language.

DESIGN TIP:
Plan metadata at ingestion — you can't filter on what you didn't store. Pre-filtering (filter then search) preserves recall better than post-filtering (search then drop). It's also the backbone of multi-tenant and permissioned RAG.

Question 11

What is hybrid search, and why is it better than pure vector search?

Accepted Answer

Hybrid search combines dense (semantic) retrieval with sparse (keyword/lexical, e.g. BM25) retrieval, then fuses their results (commonly via Reciprocal Rank Fusion). It gets the best of both: semantic matching for meaning/paraphrase, and exact matching for keywords, codes, names, and rare terms.

Why pure vector search isn't enough:
Dense embeddings can miss exact terms — product SKUs, error codes, acronyms, proper nouns, or specific numbers — because they capture meaning, not surface form. BM25 nails those exact matches but misses paraphrases. Fusing them covers each other's blind spots.

Dense results → Reciprocal Rank Fusion → merged ranking → (rerank) → top-k
BM25 results → (same fusion)

CONCRETE WIN:
"A user searching 'error TS2304' needs exact-match — dense search may return generically 'similar' errors. BM25 catches the exact code; the vector side catches the conceptual 'cannot find name' explanation. Hybrid returns both." (Very relevant to code-heavy corpora.)

Reference in assignment: outcomeschool.com/blog/how-does-hybrid-search-work.

Question 12

What is re-ranking, and how does it improve RAG retrieval quality?

Accepted Answer

Reranking is a second-stage step: after fast retrieval returns a broad candidate set (say top 30–50), a cross-encoder reranker re-scores each (query, chunk) pair jointly and reorders them, so the top few passed to the LLM are the truly most relevant. It's a precision boost on top of recall-oriented retrieval.

Bi-encoder vs cross-encoder:
Retrieval uses a bi-encoder (query and doc embedded separately) — fast and scalable but approximate. A cross-encoder feeds query + chunk together through the model, so it judges relevance far more accurately — but it's too slow to run over the whole index. So you use the two-stage pattern: cheap retrieval for recall → expensive reranking for precision on a small set.

WHY IT MATTERS:
"Retrieval is recall-optimized and noisy; reranking is precision-optimized. Adding a reranker cleans up the tail — cases where the right chunk was in the candidate set but not in the top few. It's one of the highest-ROI upgrades to a naive RAG stack." (Also directly counters 'lost in the middle,' Q15.)

Reference in assignment: outcomeschool.com/blog/how-does-a-reranker-work.

Question 13

What is query transformation in RAG (HyDE, query decomposition, step-back)?

Accepted Answer

Query transformation rewrites the user's raw query before retrieval to bridge the gap between how questions are phrased and how answers are written. Three key techniques: HyDE, query decomposition, and step-back prompting.

- HyDE (Hypothetical Document Embeddings): ask the LLM to draft a hypothetical answer to the query, then embed that (not the question) for retrieval. A fake answer looks more like the real documents than the terse question does, improving semantic match.
- Query decomposition: break a complex/multi-part question into sub-questions, retrieve for each, and combine — essential for multi-hop (Q34).
- Step-back prompting: first ask a more general/abstract version of the question to retrieve broader background, then answer the specific one grounded in that context.
- Others: query expansion (add synonyms/variants), and multi-query (generate several paraphrases and union the results).

ROOT PROBLEM THEY SOLVE:
"The 'query-document asymmetry' — a short question and a full answer live in different regions of embedding space. HyDE and friends reshape the query to look more like the target, lifting recall on hard queries."

Reference in assignment: outcomeschool.com/blog/h

Question 14

How do you handle multi-document and multi-hop questions in RAG?

Accepted Answer

Multi-hop questions need facts from several documents combined through reasoning ("Which company did the CEO of X previously found?"). A single similarity search fails because no one chunk contains the full answer. The fix is decompose → retrieve per hop → synthesize, often with iterative/agentic retrieval.

Approaches:
- Query decomposition: split into sub-questions, retrieve for each, then reason over the union.
- Iterative / agentic retrieval: retrieve, reason, then issue a follow-up query using what you just learned (ReAct-style loop).
- GraphRAG: use a knowledge graph so relationships between entities are explicit and traversable (Q18).
- Rerank + larger context so all needed pieces survive to generation.

DIAGNOSTIC TO MENTION:
"Multi-hop failures usually show up as high faithfulness but low context recall — the model answers fluently from partial context. The fix is retrieval-side (decomposition/iteration), not prompting."

Question 15

What is the "lost in the middle" problem in RAG systems?

Accepted Answer

It's the finding that LLMs use information at the beginning and end of a long context far better than in the middle (a U-shaped accuracy curve). In RAG this means that if you stuff many retrieved chunks in, the relevant one sitting in the middle can be effectively ignored — hurting answer quality even when retrieval succeeded.

RAG-specific fixes:
- Rerank and keep only the top 3–5 chunks — don't dump 20.
- Order strategically: place the most relevant chunks at the start and end of the context.
- Retrieve precisely (hybrid + rerank) so there's less filler to bury the signal.
- Restate the question after the context.

COUNTER-INTUITIVE LESSON:
"More context isn't better. Beyond a few well-chosen chunks, extra context adds noise and triggers lost-in-the-middle. Retrieval quality > retrieval quantity — that's why reranking down to a handful beats naive top-20."

Question 16

Explain Agentic RAG.

Accepted Answer

Agentic RAG wraps retrieval in an agent loop: instead of a fixed retrieve-then-read pipeline, an LLM agent decides whether to retrieve, which tool/source/index to query, reformulates queries, evaluates the results, and retrieves again if needed — reasoning and acting across multiple steps until it can answer.

What makes it "agentic":
- Dynamic retrieval: retrieve only when necessary, possibly multiple times.
- Routing: choose among several retrievers/tools (vector DB, SQL, web search, APIs).
- Query planning: decompose and iterate for multi-hop questions.
- Self-correction: judge if retrieved context is sufficient; if not, refine and retry.

WHEN TO REACH FOR IT:
Agentic RAG shines for complex, multi-step, multi-source questions — and it's more accurate but slower and costlier than naive RAG. Use naive RAG for simple lookups; go agentic when queries genuinely need planning and multiple sources. (Right in Prakhar's Rowboat agentic wheelhouse.)

Reference in assignment: outcomeschool.com/blog/agentic-rag.

Question 17

Explain Self-RAG. How does the model decide when to retrieve?

Accepted Answer

Self-RAG (Self-Reflective RAG) trains the model to control its own retrieval and to critique its own output using special reflection tokens. Crucially, it doesn't blindly retrieve on every query — it emits a Retrieve token to decide, per step, whether external knowledge is even needed.

The reflection tokens:
- Retrieve: retrieve now / no retrieval / continue with existing evidence — the on-demand decision.
- ISREL (relevance): is a retrieved passage actually relevant?
- ISSUP (support): is the generated statement supported by the passage? (catches hallucination)
- ISUSE (utility): is the overall response useful?

A critic model generates these tokens offline to build training data; the generator then learns to produce them itself, so no separate critic is needed at inference. This makes retrieval adaptive (cheaper) and adds self-checking (more factual).

KEY CONTRAST:
"Standard RAG always retrieves a fixed k, even when the model already knows the answer or the docs are irrelevant. Self-RAG decides whether and how much to trust retrieval, and verifies its own support — retrieval becomes a learned, reflective behavior, not a hard-coded step."

Question 18

What is GraphRAG, and when would you use it over traditional RAG?

Accepted Answer

GraphRAG builds a knowledge graph from your corpus — extracting entities and their relationships — and retrieves by traversing that graph (plus, in Microsoft's version, pre-computed community summaries). It captures relationships between facts that vector similarity alone can't, which is what makes it strong on multi-hop and "global" questions.

When to prefer it:
- Multi-hop reasoning across connected entities (who → founded → what → acquired-by → whom).
- Global/holistic questions over a whole corpus ("what are the main themes?") — community summaries beat scattered chunks.
- Highly interconnected domains: org charts, medical/biology, legal, supply chains, investigations.

Trade-offs:
Graph construction (entity/relation extraction) is expensive and complex to build and maintain. For simple fact lookups, vector RAG is cheaper and sufficient.

CRISP POSITIONING:
"Vector RAG finds similar text; GraphRAG finds connected facts. Reach for GraphRAG when the answer lives in the relationships between entities or requires synthesizing across the whole corpus — not for one-off lookups."

Reference in assignment: outcomeschool.com/blog/graphrag.

Question 19

How do you handle structured data (tables, SQL databases) in a RAG pipeline?

Accepted Answer

Don't force structured data through semantic chunking — that destroys its structure and precision. Instead, route it: for databases, use Text-to-SQL (the LLM generates a query that runs against the DB and returns exact rows); for tables in documents, preserve structure and query it appropriately rather than flattening it into prose.

Approaches:
- Text-to-SQL / query generation: LLM writes SQL from the natural-language question → execute → feed results back. Exact aggregates, no hallucinated numbers.
- Table-aware ingestion: extract tables as structured objects (Markdown/HTML/JSON), optionally with an LLM-generated summary embedded for retrieval while keeping the raw table for the answer.
- Row/schema embedding: embed row descriptions or schema so relevant tables surface, then run precise queries.
- Routing / agentic: a router sends numeric/aggregation questions to SQL and free-text questions to vector RAG.

KEY INSIGHT:
"LLMs are terrible at arithmetic over text-flattened tables and hallucinate numbers. For anything requiring exact values or aggregation, generate a query against structured storage — don't ask the model to eyeball a table dumped into the prompt."

Question 20

How do you evaluate a RAG system? Explain faithfulness, relevance, and context precision/recall.

Accepted Answer

You evaluate retrieval and generation separately, because a bad answer could be a retrieval failure or a generation failure. The RAGAS framework's four core metrics split cleanly: context precision & context recall assess retrieval; faithfulness & answer relevancy assess generation.

Metrics:
- Context Precision (Retrieval): Are retrieved chunks relevant & ranked well (signal vs noise)?
- Context Recall (Retrieval): Did we retrieve all the info needed to answer?
- Faithfulness (Generation): Is the answer grounded in the context (no hallucination)?
- Answer Relevancy (Generation): Does the answer actually address the question?

These are typically scored with an LLM-as-judge (reference-free) or against gold labels (precision@k, recall@k, MRR, NDCG for retrieval).

WAR STORY THAT PROVES YOU GET IT:
"I've seen faithfulness stay at 0.91 while the system was quietly wrong — because context recall was 0.62. The retriever missed a document on multi-hop queries, and the generator answered coherently from partial context, so no generation metric flagged it. Lesson: you must measure retrieval separately, or you'll miss regressions that generation metrics can't see."