Q: Explain the difference between autoregressive and masked language modeling.

Autoregressive LM (GPT) predicts the next token using only left context — great for generation. Masked LM (BERT) randomly masks tokens and predicts them using both left and right context — great for understanding/representations, but not natural generation. Comparison: - Context: Left-only (causal) for Autoregressive vs Bidirectional for Masked. - Objective: Predict next token vs Predict masked tokens. - Best at: Generation vs Understanding, embeddings, classification. - Architecture: Decoder-only vs Encoder-only. ONE-LINER: "AR is a writer (predicts what comes next); MLM is a reader (fills in blanks with full context). Generation went decoder-only; embeddings/retrieval still lean on encoder-style, bidirectional models."

Q: Difference between open-source and closed-source LLMs — when would you choose one?

Closed-source (GPT, Claude, Gemini) are accessed via API — highest capability, zero infra, but you don't control weights, data flow, or availability. Open-weight (Llama, Qwen, Mistral, DeepSeek) you can download, self-host, fine-tune, and inspect — full control and privacy, but you own the infra, quality gap, and ops. Comparison: - Capability: Usually frontier (Closed) vs Closing fast, often behind (Open-weight). - Control / customization: Limited (prompt, some FT) vs Full (weights, FT, quantize). - Data privacy: Sent to vendor vs Stays in your VPC. - Cost model: Per-token vs Fixed infra (cheaper at scale). - Ops burden: None vs You run serving/GPUs. Choose open when: data can't leave your environment (healthcare/finance), you need deep customization, you're at high volume where self-hosting is cheaper, or you need reproducibility/no vendor lock-in. Choose closed when: you want the best quality fast, low/spiky volume, small team, and no GPU ops appetite. MATURE FRAMING: "It's rarely all-or-nothing. Many production stacks route: cheap open model for easy/high-volume calls, frontier API for hard ones — optimizing cost, privacy, and quality per request." (Ties directly to Prakhar'

Q: Small Language Models (SLMs)

SLMs are compact language models (roughly ~1–10B params, often 1–3B) designed for efficiency: on-device/edge deployment, low latency, low cost, and privacy. They trade some raw capability for the ability to run locally or cheaply at scale. They punch above their weight through high-quality/curated training data (e.g. the Phi "textbooks" approach), distillation from larger teachers, and quantization. Sweet spots: on-device assistants, classification/extraction, routing, and being the cheap first tier in a model cascade. WHY THEY'RE HOT: The industry realized most production tasks don't need a frontier model. A well-tuned SLM for a narrow task is often faster, cheaper, more private, and "good enough" — and increasingly the default for agentic sub-tasks.

Q: Large Reasoning Models (LRMs)

LRMs are models explicitly trained (usually via large-scale RL) to produce an extended internal chain-of-thought before answering — spending inference-time compute to "think." Examples: OpenAI's o-series and DeepSeek-R1. They shine on math, code, and multi-step logic. The key shift is inference-time scaling: instead of only scaling training, you let the model reason longer at test time for harder problems. DeepSeek-R1 showed reasoning can emerge from pure RL (GRPO) with verifiable rewards, producing self-verification and "aha" corrections. TRADE-OFF TO STATE: LRMs cost more latency and tokens (the hidden reasoning is real compute you pay for), and are overkill for simple tasks. Route reasoning models only to genuinely hard queries; use fast models for everything else.

Q: What is the Transformer architecture and how does it work?

The Transformer (Vaswani et al., 2017, "Attention Is All You Need") is a neural architecture built entirely on self-attention instead of recurrence or convolution. It processes all tokens in parallel and lets every token directly attend to every other token, capturing long-range dependencies efficiently. How it works: Input tokens are embedded and given positional information. Then a stack of identical layers each apply: (1) multi-head self-attention to mix information across positions, and (2) a position-wise feed-forward network — each wrapped in a residual connection and normalization. The decoder adds causal masking so it can generate autoregressively. WHY IT WON: Recurrence (RNN/LSTM) is sequential and forgets long dependencies. Attention is parallelizable (GPU-friendly) and has a direct path between any two tokens, so it trains far faster and scales — which is what enabled modern LLMs.

Q: What are the key components of the Transformer architecture?

Six building blocks: (1) token embeddings, (2) positional encoding, (3) multi-head self-attention, (4) position-wise feed-forward networks, (5) residual connections, and (6) layer normalization — stacked N times, with a final linear + softmax output head. - Embeddings: tokens -> vectors. - Positional encoding: injects order (attention is order-agnostic). - Multi-head attention: the mixing engine — each token gathers relevant info from others. - Feed-forward network: per-token transformation; most parameters/knowledge live here. - Residual connections: x + sublayer(x) for gradient flow. - Normalization (LayerNorm/RMSNorm): stabilizes training. MENTAL MODEL: Each layer does two things: attention decides where to look, the FFN decides what to do with what it found. Residual + norm keep the deep stack trainable.

Question 1

What are foundation models, and how have they changed AI engineering?

Accepted Answer

A foundation model is a large model pre-trained on broad, unlabeled data at scale that can be adapted to many downstream tasks (via prompting, fine-tuning, or tools) rather than being built for one task. The term (coined by Stanford's CRFM) captures the shift from many narrow models to one general model reused everywhere.

They changed AI engineering fundamentally: previously you collected a labeled dataset and trained a bespoke model per task. Now you start from a capable pre-trained model and adapt it. The job moved from "train a model" to "orchestrate a model" — prompting, RAG, fine-tuning, evaluation, guardrails, and serving. That's exactly why "AI Engineer" exists as a role distinct from "ML Engineer/Researcher."

WHY IT MATTERS:
Foundation models democratized capability (no massive labeled data needed per task) but centralized risk — everyone inherits the same biases, failure modes, and knowledge cutoff. So evaluation and guardrails became core engineering, not an afterthought.

Question 2

What is a Large Language Model (LLM), and how does it work?

Accepted Answer

An LLM is a neural network (almost always a Transformer) with billions of parameters, trained on massive text to predict the next token given the preceding tokens. Everything it does — answering, coding, translating — is an emergent consequence of very good next-token prediction.

The pipeline:
1. Tokenize text into sub-word tokens -> integer IDs.
2. Embed each token into a vector; add positional information.
3. Transformer layers (self-attention + feed-forward) mix information across positions to build contextual representations.
4. Output head produces logits over the vocabulary -> softmax -> probability of the next token.
5. Sample a token, append it, and repeat (autoregressive generation).

Training is self-supervised: the labels come free from the text itself (the next token). Scale of data + parameters + compute yields emergent abilities like in-context learning.

THE ONE-LINER THAT LANDS:
"It's a next-token predictor. There's no database lookup or reasoning engine inside — reasoning, facts, and style are all compressed into weights and surfaced by predicting the most likely continuation."

Question 3

Inside ChatGPT: what happens after you hit Enter?

Accepted Answer

Your message is combined with the system prompt and conversation history, tokenized, and sent to the model. The model runs a prefill pass over the whole prompt (building the KV cache), then generates the reply one token at a time in a decode loop, streaming tokens back as they're produced.

Step by step:
1. Assemble context: system prompt + prior turns + your message (subject to the context window).
2. Tokenize to IDs.
3. Prefill: one forward pass processes all prompt tokens in parallel, populating the KV cache. This dominates time-to-first-token.
4. Decode loop: compute logits for the next token -> apply temperature/top-p -> sample -> append -> repeat, reusing the KV cache, until an end-of-sequence token or length limit.
5. Stream + post-process: detokenize, run safety filters, stream to your screen.

DEPTH SIGNAL:
Mention prefill (compute-bound, parallel) vs decode (memory-bandwidth-bound, sequential). That distinction explains latency behavior and why the first token feels slow (see Q33).

Question 4

What are autoregressive models?

Accepted Answer

An autoregressive (AR) model generates a sequence one element at a time, where each new token is conditioned on all previously generated tokens. It factorizes the joint probability of a sequence into a product of left-to-right conditionals: P(x) = product of P(xt | x<t).

GPT-style LLMs are decoder-only autoregressive models. This is why generation is inherently sequential (you can't produce token t+1 before t) — which is also the root of decode-phase latency and why techniques like speculative decoding and diffusion LMs try to parallelize it.

TRADE-OFF:
AR gives coherent, high-quality generation but is sequential (slow) and struggles with tasks needing bidirectional context or "planning backward" (the reversal curse).

Question 5

Explain the difference between autoregressive and masked language modeling.

Accepted Answer

Autoregressive LM (GPT) predicts the next token using only left context — great for generation. Masked LM (BERT) randomly masks tokens and predicts them using both left and right context — great for understanding/representations, but not natural generation.

Comparison:
- Context: Left-only (causal) for Autoregressive vs Bidirectional for Masked.
- Objective: Predict next token vs Predict masked tokens.
- Best at: Generation vs Understanding, embeddings, classification.
- Architecture: Decoder-only vs Encoder-only.

ONE-LINER:
"AR is a writer (predicts what comes next); MLM is a reader (fills in blanks with full context). Generation went decoder-only; embeddings/retrieval still lean on encoder-style, bidirectional models."

Question 6

Difference between open-source and closed-source LLMs — when would you choose one?

Accepted Answer

Closed-source (GPT, Claude, Gemini) are accessed via API — highest capability, zero infra, but you don't control weights, data flow, or availability. Open-weight (Llama, Qwen, Mistral, DeepSeek) you can download, self-host, fine-tune, and inspect — full control and privacy, but you own the infra, quality gap, and ops.

Comparison:
- Capability: Usually frontier (Closed) vs Closing fast, often behind (Open-weight).
- Control / customization: Limited (prompt, some FT) vs Full (weights, FT, quantize).
- Data privacy: Sent to vendor vs Stays in your VPC.
- Cost model: Per-token vs Fixed infra (cheaper at scale).
- Ops burden: None vs You run serving/GPUs.

Choose open when:
data can't leave your environment (healthcare/finance), you need deep customization, you're at high volume where self-hosting is cheaper, or you need reproducibility/no vendor lock-in.

Choose closed when:
you want the best quality fast, low/spiky volume, small team, and no GPU ops appetite.

MATURE FRAMING:
"It's rarely all-or-nothing. Many production stacks route: cheap open model for easy/high-volume calls, frontier API for hard ones — optimizing cost, privacy, and quality per request." (Ties directly to Prakhar'

Question 7

Small Language Models (SLMs)

Accepted Answer

SLMs are compact language models (roughly ~1–10B params, often 1–3B) designed for efficiency: on-device/edge deployment, low latency, low cost, and privacy. They trade some raw capability for the ability to run locally or cheaply at scale.

They punch above their weight through high-quality/curated training data (e.g. the Phi "textbooks" approach), distillation from larger teachers, and quantization. Sweet spots: on-device assistants, classification/extraction, routing, and being the cheap first tier in a model cascade.

WHY THEY'RE HOT:
The industry realized most production tasks don't need a frontier model. A well-tuned SLM for a narrow task is often faster, cheaper, more private, and "good enough" — and increasingly the default for agentic sub-tasks.

Question 8

Large Reasoning Models (LRMs)

Accepted Answer

LRMs are models explicitly trained (usually via large-scale RL) to produce an extended internal chain-of-thought before answering — spending inference-time compute to "think." Examples: OpenAI's o-series and DeepSeek-R1. They shine on math, code, and multi-step logic.

The key shift is inference-time scaling: instead of only scaling training, you let the model reason longer at test time for harder problems. DeepSeek-R1 showed reasoning can emerge from pure RL (GRPO) with verifiable rewards, producing self-verification and "aha" corrections.

TRADE-OFF TO STATE:
LRMs cost more latency and tokens (the hidden reasoning is real compute you pay for), and are overkill for simple tasks. Route reasoning models only to genuinely hard queries; use fast models for everything else.

Question 9

What is the Transformer architecture and how does it work?

Accepted Answer

The Transformer (Vaswani et al., 2017, "Attention Is All You Need") is a neural architecture built entirely on self-attention instead of recurrence or convolution. It processes all tokens in parallel and lets every token directly attend to every other token, capturing long-range dependencies efficiently.

How it works:
Input tokens are embedded and given positional information. Then a stack of identical layers each apply: (1) multi-head self-attention to mix information across positions, and (2) a position-wise feed-forward network — each wrapped in a residual connection and normalization. The decoder adds causal masking so it can generate autoregressively.

WHY IT WON:
Recurrence (RNN/LSTM) is sequential and forgets long dependencies. Attention is parallelizable (GPU-friendly) and has a direct path between any two tokens, so it trains far faster and scales — which is what enabled modern LLMs.

Question 10

What are the key components of the Transformer architecture?

Accepted Answer

Six building blocks: (1) token embeddings, (2) positional encoding, (3) multi-head self-attention, (4) position-wise feed-forward networks, (5) residual connections, and (6) layer normalization — stacked N times, with a final linear + softmax output head.

- Embeddings: tokens -> vectors.
- Positional encoding: injects order (attention is order-agnostic).
- Multi-head attention: the mixing engine — each token gathers relevant info from others.
- Feed-forward network: per-token transformation; most parameters/knowledge live here.
- Residual connections: x + sublayer(x) for gradient flow.
- Normalization (LayerNorm/RMSNorm): stabilizes training.

MENTAL MODEL:
Each layer does two things: attention decides where to look, the FFN decides what to do with what it found. Residual + norm keep the deep stack trainable.

Question 11

Encoder-only vs decoder-only vs encoder-decoder — what's the difference?

Accepted Answer

They differ in attention direction and use case. Encoder-only (BERT) uses bidirectional attention -> understanding. Decoder-only (GPT/Llama) uses causal attention -> generation. Encoder-decoder (T5, original Transformer) encodes an input bidirectionally then decodes an output -> seq-to-seq tasks like translation.

Comparison:
- Encoder-only: Attention - Bidirectional. Example - BERT. Best for - Classification, embeddings, NER.
- Decoder-only: Attention - Causal. Example - GPT, Llama. Best for - Generation, chat (dominant for LLMs).
- Encoder-decoder: Attention - Both + cross-attn. Example - T5, BART. Best for - Translation, summarization.

WHY DECODER-ONLY DOMINATES LLMS:
It's simplest, scales cleanly, and a single next-token objective unifies every task as text-to-text — so one model does everything with in-context learning, no task-specific heads.

Question 12

What are embeddings?

Accepted Answer

An embedding is a dense, learned vector that represents a discrete item (a token, word, sentence, image) in a continuous space where semantic similarity corresponds to geometric closeness. Similar meanings land near each other; relationships become directions.

In an LLM, the embedding layer maps each token ID to a vector the network can compute over. Beyond that, "embeddings" also refers to whole-text vectors from an embedding model — the backbone of semantic search and RAG (retrieve by cosine similarity).

CONNECT TO YOUR WORK:
This is exactly the mechanism behind CareerLift's semantic job matching and any RAG system — encode text to vectors, then retrieve by nearest-neighbor similarity. Mentioning that shows applied understanding.

Question 13

What is positional encoding, and why is it needed in Transformers?

Accepted Answer

Self-attention is permutation-invariant — it treats the input as a set, with no inherent notion of order. Positional encoding injects information about each token's position so the model knows "the cat sat" ≠ "sat the cat."

Variants:
- Sinusoidal (original): fixed sine/cosine patterns of different frequencies; no parameters, extrapolates somewhat.
- Learned absolute: a trainable vector per position (BERT/GPT-2); simple but capped at trained length.
- RoPE / relative: encodes relative position; better long-range and extrapolation (see Q14). Now standard.

KEY INSIGHT:
Without positional info, a Transformer literally cannot tell word order apart. How you encode position also determines how well the model handles sequences longer than it saw in training.

Question 14

How does RoPE work, and why is it preferred over learned positional embeddings?

Accepted Answer

Rotary Position Embedding (RoPE) encodes position by rotating the query and key vectors by an angle proportional to their position before computing attention. Because attention uses dot products, the rotation makes the score depend on the relative offset between two tokens — position is baked directly into the attention mechanism.

Why it's preferred:
- Relative by construction: what matters is the distance between tokens, which generalizes better than absolute positions.
- Extrapolation: handles longer sequences than seen in training far better than learned absolute embeddings (and can be extended via interpolation/NTK scaling).
- No extra parameters and applied at each layer's attention, not just the input.

This is why RoPE (or its cousins) is used in Llama, Qwen, Mistral, and most modern LLMs. (Note: RoPE decay is also part of why long-context "lost in the middle" happens.)

ONE-LINER:
"RoPE turns absolute position into relative position via rotation, so attention scores naturally depend on token distance — that's what gives modern LLMs their long-context flexibility."

Question 15

Explain Query (Q), Key (K), and Value (V) in attention.

Accepted Answer

Q, K, V are three learned linear projections of each token's representation. Intuition: the Query is what a token is looking for, the Key is what each token offers, and the Value is the actual content passed along. Attention matches queries to keys to decide how much of each value to pull in.

Attention(Q, K, V) = softmax( Q·K^T / sqrt(dk) ) · V

A token's query is dotted with every token's key to get relevance scores; softmax turns them into weights; the output is the weighted sum of values. Q, K, V come from the same sequence in self-attention (Q from one, K/V from another in cross-attention).

ANALOGY THAT STICKS:
Library search: your query is what you want, book titles/labels are keys, book contents are values. You compare your query to every title, then read the contents of the best matches — weighted by how relevant each was.

Question 16

What is self-attention, and how does it work in Transformers?

Accepted Answer

Self-attention lets each token in a sequence attend to every other token (including itself) to build a context-aware representation. "Self" means Q, K, and V all come from the same sequence. It's how the model resolves references and gathers relevant context.

Mechanics:
1. Project each token into Q, K, V.
2. Score = Q·K^T / sqrt(dk) — every token vs every token.
3. Softmax the scores into attention weights.
4. Output = weighted sum of Values.

Example: in "The animal didn't cross the street because it was tired," self-attention lets "it" attend strongly to "animal," resolving the reference.

COST CAVEAT:
Every-token-vs-every-token is O(n²) in sequence length — the root of the context-window and long-document cost problems (see Q29, Q35, Q57).

Question 17

What is cross-attention in Transformers?

Accepted Answer

Cross-attention is attention where the Query comes from one sequence and the Keys/Values come from another. It's how a decoder conditions on an encoder's output — e.g. the translation decoder attending to the encoded source sentence.

Contrast with self-attention (all from one sequence). Cross-attention appears in encoder-decoder models (T5, translation) and multimodal models (text tokens attending to image features). It's the mechanism that lets one modality/sequence "read" another.

WHERE YOU'LL SEE IT TODAY:
Vision-language models and diffusion image models use cross-attention to condition generation on a text prompt. Decoder-only LLMs mostly use self-attention only (they concatenate everything into one sequence instead).

Question 18

Why do we scale dot-product attention by sqrt(dk)?

Accepted Answer

Because dot products grow in magnitude with dimension. For vectors of dimension dk with unit-variance components, the dot product has variance ~dk. Large scores push softmax into saturated regions where one value ≈ 1 and the rest ≈ 0, giving vanishingly small gradients. Dividing by sqrt(dk) normalizes the variance back to ~1, keeping softmax in a well-behaved, trainable range.

Var(Q·K) ≈ dk -> divide by sqrt(dk) -> Var ≈ 1

CRISP VERSION:
"Without scaling, high-dimensional dot products get huge, softmax saturates, gradients vanish, and training stalls. sqrt(dk) keeps the score variance ~1 so gradients stay healthy." Interviewers love that you can state why sqrt(dk) specifically (it's the standard deviation).

Question 19

What is causal masking?

Accepted Answer

Causal (or "look-ahead") masking prevents a token from attending to future tokens during autoregressive generation. Before softmax, the attention scores for future positions are set to -infinity (an upper-triangular mask), so their weights become zero.

This enforces that the prediction for position t depends only on positions ≤ t — essential for a decoder that must generate left-to-right and must not "cheat" by peeking at the answer during training. Encoders (BERT) omit this mask because they want bidirectional context.

WHY IT MATTERS AT TRAINING:
It lets the model be trained on a whole sequence in parallel while still learning proper next-token prediction — every position is supervised at once, but none can see its own future.

Question 20

What are multi-head attention mechanisms? Why use multiple heads?

Accepted Answer

Multi-head attention runs several attention operations ("heads") in parallel, each with its own learned Q/K/V projections into a lower-dimensional subspace. Their outputs are concatenated and linearly projected. Multiple heads let the model attend to different relationships and positions simultaneously.

One head might track syntactic dependencies, another coreference, another local vs long-range relations. A single head would have to average all these into one attention pattern; splitting into heads gives the model multiple "representation subspaces" to specialize in — richer, more expressive attention at the same total cost.

EFFICIENCY LINK:
Each head has its own K/V, which bloats the KV cache. That's exactly what Grouped-Query Attention (Q36) optimizes — sharing K/V across heads to shrink memory with minimal quality loss.

LLM Fundamentals

LLM Fundamentals