Question 1

What is fine-tuning, and when should you fine-tune an LLM?

Accepted Answer

Fine-tuning continues training a pre-trained model on a smaller, curated, task- or domain-specific dataset to specialize its behavior, style, or skills. You're not teaching it language from scratch — you're nudging an already-capable model toward doing one thing consistently well.

When to fine-tune
- Consistent behavior/format/tone that prompting can't reliably enforce.
- A specialized skill or narrow task where you have good labeled data.
- Domain adaptation (legal/medical/finance style & vocabulary).
- Latency/cost reduction: bake behavior in so you can drop long few-shot prompts.

When NOT to
Don't fine-tune to inject frequently-changing facts — that's RAG's job. And don't reach for it before exhausting prompting; fine-tuning is slower, costlier, and harder to update.

ONE-LINER
"Fine-tuning changes how a model behaves, not what it knows at runtime. Use it for form, style, and skill — not for knowledge that changes, which belongs in RAG."

Question 2

Explain the difference between full fine-tuning and parameter-efficient fine-tuning (PEFT).

Accepted Answer

Full fine-tuning updates all of the model's weights; PEFT freezes the base model and trains only a small number of extra or selected parameters (typically ~0.1–1% of the total) — methods like LoRA, adapters, and prefix/prompt tuning.

Full fine-tuning vs PEFT
- Params trained: Full FT = All (100%) | PEFT = ~0.1–1%
- Compute/memory: Full FT = Very high (grads + optimizer states for all) | PEFT = Low — fits on modest GPUs
- Artifact per task: Full FT = Full model copy (GBs) | PEFT = Tiny adapter (MBs)
- Forgetting risk: Full FT = Higher | PEFT = Lower (base frozen)
- Max adaptation: Full FT = Highest capacity | PEFT = Slightly less, usually enough

WHY PEFT DOMINATES
"PEFT gets ~95%+ of full fine-tuning's quality for a fraction of the cost, produces swappable MB-sized adapters, and forgets less because the base is frozen. It's the default now — you only reach for full fine-tuning when you need a large behavior/knowledge shift and have the compute."

Reference in assignment: "AI Engineering Explained" video.

Question 3

When should you choose fine-tuning over RAG over prompt engineering?

Accepted Answer

These are a ladder of increasing cost and commitment. Prompt engineering first (no training, instant iteration). RAG when you need external, fresh, or private knowledge with citations. Fine-tuning when you need consistent behavior/format/skill or lower latency. They combine.

Prompting (behavior via instructions) → RAG (knowledge, updatable) → Fine-tuning (behavior baked into weights)

Need → Use
- Quick iteration, simple task → Prompt engineering
- Fresh / private / large knowledge, citations → RAG
- Consistent format/tone/skill, shorter prompts → Fine-tuning
- Domain-grounded assistant → Fine-tune + RAG together

DECISION RULE
"Knowledge → RAG. Behavior → fine-tune. Speed of iteration → prompting. Start at the cheapest rung and only climb when it can't meet the bar. The mistake is fine-tuning to add facts — expensive, un-updatable, and RAG does it better."

Question 4

What is the difference between SFT (Supervised Fine-Tuning) and alignment training?

Accepted Answer

SFT trains on (input, ideal-output) pairs — pure imitation: "here's a good answer, reproduce this behavior." Alignment (RLHF/RLAIF/DPO) goes further, optimizing on preferences — "given several valid answers, which is better?" — instilling helpfulness, honesty, and harmlessness that imitation alone can't capture.

Pretrain → SFT (teach good outputs) → Alignment (teach preferred outputs / values)

SFT teaches the model to produce a good answer and follow instructions; alignment teaches it to pick the better of two acceptable answers and to internalize values. SFT uses demonstrations; alignment uses comparison/preference data.

CRISP CONTRAST
"SFT imitates demonstrations; alignment optimizes preferences. SFT gets you a competent assistant; alignment gets you a helpful, honest, harmless one. That's why the modern recipe is pretrain → SFT → alignment, not SFT alone."

Question 5

What is LoRA (Low-Rank Adaptation), and how does it work?

Accepted Answer

LoRA freezes the pre-trained weights and injects a low-rank update into weight matrices: instead of learning a full update ΔW, it learns two small matrices B and A whose product approximates it. Only A and B are trained. It rests on the finding that fine-tuning updates have low intrinsic rank — you don't need a full-rank change.

W′ = W + ΔW = W + (α/r)·B·A   where B ∈ ℝd×r, A ∈ ℝr×k, r ≪ d

The rank r (e.g. 8–64) controls capacity; α is a scaling factor. You train ~0.1–1% of parameters, producing MB-sized adapters you can swap per task. At inference, B·A can be merged into W, so there's zero added latency.

WHY IT'S THE DEFAULT PEFT
"LoRA gets near-full-fine-tuning quality by exploiting that weight updates are low-rank. Benefits: tiny adapters, hot-swappable per task, no inference overhead when merged, and less forgetting because the base is frozen. Rank and alpha are the knobs to tune." (Directly relevant to any lightweight model-adaptation work.)

Reference in assignment: outcomeschool.com/blog/lora-low-rank-adaptation-of-llms.

Question 6

What is QLoRA, and how does it enable fine-tuning on consumer hardware?

Accepted Answer

QLoRA = LoRA on top of a 4-bit quantized frozen base model. It loads the base weights in 4-bit to slash memory, then trains LoRA adapters in 16-bit, backpropagating through the frozen 4-bit weights into the adapters. This lets you fine-tune huge models (e.g. a 65B model on a single 48GB GPU) with negligible quality loss vs 16-bit LoRA.

Three key innovations
- NF4 (4-bit NormalFloat): a quantization data type that's information-theoretically optimal for the normally-distributed weights of neural nets — better than plain int4.
- Double quantization: quantize the quantization constants themselves, saving additional memory.
- Paged optimizers: use NVIDIA unified memory to gracefully handle memory spikes (avoids OOM crashes).

WHY IT MATTERED
"QLoRA democratized fine-tuning — it made billion-parameter models tunable on a single consumer/prosumer GPU. The base stays frozen in 4-bit (memory win); only the tiny adapters train in higher precision (quality preserved). That's how a student with one GPU can fine-tune a large model." (Very relevant for Prakhar's independent-builder setup.)

Reference in assignment: "AI Engineering Explained" video.

Question 7

Explain Prefix Tuning and Prompt Tuning. How are they different from LoRA?

Accepted Answer

Both learn continuous "soft" vectors (virtual tokens) that steer the frozen model, without touching its weights. Prompt tuning prepends learned vectors to the input embeddings only (input layer). Prefix tuning prepends learned vectors to the keys/values at every layer — more expressive. LoRA, by contrast, modifies the weight matrices via low-rank updates.

Method → What it learns → Where
- Prompt tuning → Soft prompt vectors → Input embeddings only
- Prefix tuning → Prefix vectors (K/V) → Every layer
- LoRA → Low-rank weight updates → Weight matrices

vs LoRA: prompt/prefix tuning steer activations via virtual tokens; LoRA changes the weights. Prompt tuning is the most parameter-efficient (fewest params) but generally less powerful; LoRA is usually more performant and can be merged into weights (no inference overhead), whereas soft prompts consume context tokens.

ONE-LINER
"Prompt/prefix tuning add learned virtual tokens (steer the input/activations); LoRA rewrites the weights with a low-rank delta. LoRA is more expressive and merge-able; prompt tuning is the leanest. Same goal — adapt a frozen model — different lever."

Question 8

What is adapter-based fine-tuning?

Accepted Answer

Adapter tuning inserts small trainable bottleneck modules (down-project → nonlinearity → up-project) inside each Transformer layer, freezes everything else, and trains only the adapters. It was the original PEFT method (Houlsby et al., 2019) and, like LoRA, gives modular, swappable, few-parameter adaptation.

Adapter(x) = x + Wup·σ(Wdown·x)   (bottleneck: d → small → d)

vs LoRA: adapters add new layers in the forward pass, so they introduce inference latency; LoRA is a reparameterization of existing weights that can be merged in, adding no latency. That's a big reason LoRA overtook classic adapters for LLMs.

FRAMING
"Adapters were the proof-of-concept for PEFT — freeze the base, train tiny modules. LoRA improved on the idea by making the update mergeable into the weights, eliminating the extra-layer latency. Both are modular; LoRA won on inference efficiency."

Reference in assignment: "AI Engineering Explained" video.

Question 9

How do you merge multiple LoRA adapters?

Accepted Answer

You can combine several task-specific adapters into one multi-task model by fusing their weight deltas ("task vectors"). The naive method is weight averaging; better methods (TIES, DARE) reduce the interference that arises when adapters conflict.

Methods
- Weight-space averaging: simple mean of the adapter weights — baseline, but adapters can cancel/interfere.
- TIES-merging: Trim redundant (small) parameter changes → Elect a consistent sign (resolve sign conflicts) → Merge only same-sign params. Reduces destructive interference.
- DARE: Drop a random fraction of the delta params And REscale the rest per task vector, then fuse — exploits that most delta params are redundant.

KEY CONCEPT
"Think of each adapter as a 'task vector' added to the base. Naive averaging causes interference; TIES and DARE prune and sign-align the deltas so tasks don't cancel each other. This lets you build a multi-skill model by merging single-skill adapters — no retraining."

Reference in assignment: outcomeschool.com/blog/lora-low-rank-adaptation-of-llms.

Question 10

What is RLHF, and how is it used to align LLMs?

Accepted Answer

RLHF aligns a model to human preferences — helpfulness, honesty, harmlessness — that are hard to write as a loss function. The classic three-stage recipe: (1) SFT on demonstrations, (2) train a reward model on human preference comparisons, (3) optimize the policy with RL (PPO) against that reward, with a KL penalty to stay near the SFT model.

1. SFT: fine-tune on high-quality human-written responses.
2. Reward model: humans rank pairs of responses; a model learns to predict the preferred one → scalar reward.
3. RL optimization: the policy generates, the reward model scores, PPO updates — KL-penalized so it doesn't drift into reward-gaming gibberish.

WHY PIVOTAL
"RLHF is what turned raw next-token predictors (GPT-3) into aligned assistants (ChatGPT). It optimizes 'what humans prefer,' which pure likelihood training can't capture. DPO and RLAIF are newer, cheaper ways to do the preference-optimization stage."

Reference in assignment: outcomeschool.com/blog/reinforcement-learning-from-human-feedback-rlhf.

Question 11

What is RLAIF (RL from AI Feedback), and how does it differ from RLHF?

Accepted Answer

RLAIF replaces the human preference labelers with an AI model that judges which output is better — typically against a set of written principles (a "constitution," as in Anthropic's Constitutional AI). The rest of the pipeline (reward modeling / preference optimization + RL) stays the same; only the source of the preference signal changes.

Constitutional AI's two phases
- Supervised (self-critique): the model critiques and revises its own harmful outputs against the principles; fine-tune on the revisions.
- RLAIF: the model generates preference pairs, an AI judge ranks them per the constitution, and that trains the reward model / drives RL (or DPO).

RLHF vs RLAIF
- Preference labeler: RLHF = Humans | RLAIF = An AI model + principles
- Cost / scale: RLHF = Slow, expensive | RLAIF = Cheap, highly scalable
- Consistency: RLHF = Varies by annotator | RLAIF = Same rules every time; transparent

NUANCE THAT IMPRESSES
"Google showed RLAIF roughly matches RLHF quality at a fraction of the labeling cost. The risk: if the AI judge shares the generator's blind spots, RLAIF can reinforce a shared error — so it's one layer in a stack (red-teaming, classifiers, policies), not a complete alignm

Question 12

What is instruction tuning, and why is it important for chat models?

Accepted Answer

Instruction tuning is SFT on a diverse set of (instruction, response) pairs across many tasks, teaching the model to follow natural-language instructions rather than merely continue text. It's what converts a raw base model (a next-token predictor) into a helpful assistant that responds to commands and questions.

A base model completes text ("The capital of France is…"); an instruction-tuned model responds to intent ("What's the capital of France?" → "Paris."). By training on many instruction types, the model also learns to generalize to unseen instructions zero-shot. Examples: FLAN, InstructGPT.

WHY IT'S THE FOUNDATION OF CHAT
"Instruction tuning is the bridge from 'text completer' to 'assistant.' Without it, a base model doesn't reliably answer questions or follow commands — it just predicts plausible continuations. It's the first alignment step, usually followed by preference alignment (RLHF/RLAIF)."

Question 13

How do you prepare a dataset for fine-tuning an LLM?

Accepted Answer

Dataset quality is the single biggest driver of fine-tuning success — far more than quantity. The process: define the exact task and output format, collect high-quality (input, output) examples that are diverse and cover edge cases, clean and deduplicate, format consistently to the model's chat template, balance classes, and split into train/validation/test.

Checklist
- Quality > quantity: a few hundred–thousand clean examples beat tens of thousands of noisy ones.
- Diversity & coverage: represent the real input distribution, including edge cases.
- Clean & dedupe: remove errors, near-duplicates (which cause memorization), and label noise.
- Consistent formatting: use the correct chat template / special tokens.
- Held-out split the model never trains on, for honest eval.

THE RULE
"Garbage in, garbage out — fine-tuning amplifies whatever's in your data, including its errors and biases. I'd spend most of the effort curating and validating data, not tweaking hyperparameters. Clean, diverse, well-formatted data is the whole game."

Question 14

What is synthetic data generation, and how do you use it for fine-tuning?

Accepted Answer

Synthetic data is training data generated by an LLM (or templates/rules) rather than collected/labeled by humans. It's used to overcome data scarcity — augmenting small datasets, covering edge cases, and creating instruction/preference pairs — often by having a stronger "teacher" model produce examples.

Techniques
- Self-Instruct: the model generates its own instructions + responses to bootstrap an instruction dataset.
- Evol-Instruct: iteratively evolve prompts to be harder/more diverse.
- Teacher generation: a stronger model produces high-quality examples (a form of distillation).
- Filtering/verification: keep only correct, high-quality samples (critical).

RISKS TO NAME
"Synthetic data scales cheaply but propagates the generator's biases and errors, and over-reliance risks 'model collapse.' Always filter/verify it, and mind the legal angle — generating training data from a proprietary model's API can violate its Terms of Service (see Q18)."

Question 15

What are the key hyperparameters for fine-tuning?

Accepted Answer

The ones that matter most: learning rate (the single most important), epochs, batch size, and — for LoRA — rank (r) and alpha.

Hyperparameter → Effect / typical range
- Learning rate → Most impactful. Too high → instability/forgetting; too low → underfit. LoRA uses higher LR (~1e-4–3e-4) than full FT (~1e-5–5e-5).
- Epochs → Usually 1–3. Too many → overfitting/memorization.
- Batch size → Affects stability & memory; use gradient accumulation for a larger effective batch.
- LoRA rank (r) → Capacity of the adaptation (8–64 common). Higher = more capacity + params.
- LoRA alpha → Scaling; effective scale = α/r (often α ≈ 2r).
- Target modules → Which layers get LoRA (attn q/v/k/o, sometimes MLP).

PRACTICAL ORDER
"Get the learning rate right first, keep epochs low (1–3) to avoid memorization, then tune LoRA rank for capacity. Watch validation loss — if train loss keeps dropping while val rises, you're overfitting, so pull back epochs/LR."

Reference in assignment: outcomeschool.com/blog/lora-low-rank-adaptation-of-llms.

Question 16

How do you fine-tune a model for a specific domain (legal, medical, finance)?

Accepted Answer

Layer your approach by how deep the domain shift is: try prompting/RAG first; then instruction fine-tune on domain (instruction, response) pairs for domain tasks and style; and for a strong domain shift, continual pre-train on a raw domain corpus to inject vocabulary and knowledge before instruction tuning.

The stack
1. RAG + prompting — cheapest; often enough for factual grounding.
2. Instruction FT on curated domain examples — adapts task behavior and style.
3. Continual pre-training on domain text (Q17) — injects deep domain knowledge/jargon.
4. Domain tokenizer if jargon fragments badly.

Domain-specific care
These are high-stakes domains: use expert-validated data, add safety guardrails and human-in-the-loop, ground facts with RAG to curb hallucination, and respect regulatory/compliance constraints.

BEST-PRACTICE COMBO
"For a domain assistant I'd combine continual pre-training (knowledge) + instruction FT (behavior) + RAG (fresh, citable facts). In medical/legal/finance, correctness is safety-critical — so expert validation, guardrails, and grounding aren't optional."

Question 17

What is continual pre-training, and when would you use it?

Accepted Answer

Continual (or continued) pre-training keeps running the self-supervised next-token objective on a large new/domain corpus before task fine-tuning. Unlike SFT (which teaches behavior on labeled pairs), it injects domain knowledge, vocabulary, and style at the representation level from raw, unlabeled text.

When to use it
- Strong domain shift: legal, medical, biomedical, code, finance.
- A new language or specialized vocabulary the base handles poorly.
- You have lots of unlabeled domain text but limited labeled data.

DISTINCTION TO STATE
"Continual pre-training adds knowledge; SFT adds behavior. Use continual pretrain when the base model doesn't 'speak' your domain, then instruction-tune on top. Watch catastrophic forgetting — mix in some general data and use a modest learning rate so you don't wipe out general ability."

Question 18

What is knowledge distillation for fine-tuning, and what are the legal considerations?

Accepted Answer

Distillation trains a smaller "student" on a larger "teacher's" outputs — either soft labels/logits or, more commonly for LLMs, generated training data — to create a cheaper model that mimics the teacher's capability. It's a standard, powerful technique. The legal catch is where the teacher's outputs come from.

The legal landscape
- ToS restrictions: distilling from a proprietary model via its API typically violates that provider's Terms of Service — OpenAI's and Anthropic's terms expressly forbid using outputs to build competing/"imitation" models.
- The precedent: the OpenAI v. DeepSeek dispute (2025) — OpenAI alleged DeepSeek used its outputs to train R1, breaching ToS; Anthropic later reported large-scale unauthorized extraction attempts.
- ToS ≠ IP: a ToS breach is a contract issue, not clear-cut copyright/IP infringement — enforcement is uncertain and the burden of proof is on the accuser.
- License matters: if the teacher's license permits distillation (permissively-licensed open models — MIT/Apache, or Llama within its terms), it's perfectly fine.

SAFE PRACTICE
"Distillation itself is legitimate and standard. The rule of thumb: check the teacher model's license/ToS before

Question 19

What is catastrophic forgetting, and how do you prevent it during fine-tuning?

Accepted Answer

Catastrophic forgetting is when fine-tuning on new data erases or degrades previously learned capabilities — the model gets better at the new task but worse at general skills, because gradient updates overwrite the weights that encoded them.

Prevention
- PEFT (LoRA/adapters): freezing the base weights inherently preserves prior knowledge — the biggest lever.
- Lower learning rate + fewer epochs: smaller, gentler updates.
- Replay / rehearsal: mix in general or original-distribution data during fine-tuning.
- Regularization (EWC): penalize changes to weights important for old tasks.
- Evaluate on general benchmarks to detect forgetting early.

BEST DEFAULT
"The simplest strong defense is LoRA — the base is frozen, so there's little to forget. If you must full-fine-tune, mix in general data (replay) and keep the learning rate modest. And always benchmark general ability before/after, or you won't notice the regression."

Reference in assignment: outcomeschool.com/blog/continual-learning-in-llms.

Question 20

How do you evaluate a fine-tuned model's performance?

Accepted Answer

Evaluate on a held-out set the model never trained on, with task-appropriate metrics, and — critically — compare against the right baselines (the base model + prompting) to prove the fine-tuning actually helped. Also check for regressions and overfitting.

What to measure
- Task metrics: accuracy/F1/exact-match for classification; BLEU/ROUGE or LLM-as-judge for generation.
- Baseline comparison: vs base model and vs prompting — did FT beat the cheaper option?
- Regression check: general benchmarks to catch catastrophic forgetting.
- Overfitting: train vs validation gap; verbatim-recall tests.
- Safety/bias eval, then real-world A/B.

THE TRAP TO AVOID
"The #1 mistake is evaluating only on the new task and declaring victory — you have to (a) test on data it never saw, (b) beat the prompting baseline to justify the cost, and (c) confirm it didn't forget its general skills. Fine-tuning that wins the task but breaks everything else isn't a win."

Reference in assignment: outcomeschool.com/blog/llm-evaluation.

Fine-Tuning & Model Adaptation

Fine-Tuning & Model Adaptation