Back to Tracker
Phase 2 — LLMs & Transformers

Transformer Architecture & Large Language Models

Every LLM — GPT-4, Claude, Gemini, Llama — runs on this. Know attention cold. Know why hallucination happens. These are the most asked topics in 2026 SDE AI interviews.

Attention Q/K/V Multi-Head Attention Encoder vs Decoder Tokenization (BPE) Context Window LoRA / PEFT Hallucination
Why Transformers Replaced RNNs
RNNs (Old Approach) — Problems
  • Process tokens one at a time — cannot parallelize training
  • Long-range dependencies decay — model "forgets" what was early in the sequence
  • Training is very slow (sequential dependency)
  • Vanishing/exploding gradient problem over long sequences
Transformers — Solutions
  • Process all tokens simultaneously using attention — fully parallelizable
  • Direct connections between any two tokens — no forgetting
  • Scales to trillions of tokens on thousands of GPUs
  • Residual connections prevent vanishing gradients
The Attention Mechanism: Q, K, V CRITICAL

Attention answers one question: "For each token, how much should I pay attention to every other token?"

Q
Query

"What information am I looking for?"

Each token asks a question about what context it needs

K
Key

"What do I represent / offer?"

Each token broadcasts what information it contains

V
Value

"What information do I actually contribute?"

The actual content passed to tokens that attend to me

The Formula — Know It Cold

Attention(Q, K, V) = softmax( Q × K^T / √d_k ) × V

Step by Step

  1. Q × K^T — dot product gives raw similarity scores between every pair of tokens
  2. / √d_k — scale to prevent softmax from saturating in high dimensions
  3. softmax() — convert scores to probabilities (attention weights, sum to 1)
  4. × V — weighted sum of all token values

Result

Each token's output is a weighted combination of all tokens' values, where the weights reflect how relevant each token is to the current token.

The model learns what to attend to during training — no manual rules. Different heads learn different relationship types (syntax, semantics, coreference).
Multi-Head Attention HIGH

Instead of one attention computation, run it h times in parallel (h = number of heads, typically 8–32).

Why Multiple Heads?

Each head learns a different type of relationship between tokens:

  • Head 1 might focus on syntactic dependencies (subject → verb)
  • Head 2 might focus on coreference ("it" → "the dog")
  • Head 3 might focus on semantic similarity
  • Head 4 might focus on positional proximity

The results from all heads are concatenated and projected back to the original dimension.

Architecture Note
GPT-2 Small12 heads, d_model=768
GPT-396 heads, d_model=12288
Each head dimd_model / num_heads
OutputConcat all heads → linear projection → d_model
The Full Transformer Block — Every Component Explained
ComponentWhat It DoesWhy It's Needed
Token EmbeddingConverts token IDs (integers) to dense vectorsGives the model a learnable numeric representation for each vocabulary item
Positional EncodingAdds position information to each embeddingAttention has no built-in sense of order — without this, "dog bites man" = "man bites dog"
Multi-Head Self-AttentionEach token attends to all other tokens (decoder: only past)Captures relationships between any two tokens, regardless of distance
Feed-Forward Network (FFN)Two linear layers with ReLU/GELU in between, applied per token independentlyWhere "factual knowledge" is stored — acts as a key-value memory for facts
Layer NormalizationNormalize activations to mean=0, std=1 within each layerStabilizes training — prevents activations from exploding or vanishing
Residual ConnectionsAdd the input directly to the output: output = f(x) + xCreates gradient highways — gradients can flow directly to early layers, enabling very deep networks
Encoder vs Decoder — BERT vs GPT HIGH
Encoder-Only (BERT, RoBERTa)

Bidirectional — each token attends to ALL tokens (past and future)

Training task: masked language modeling (predict masked tokens)

Best for:

  • Sentence classification (sentiment, intent)
  • Semantic search (generate embeddings)
  • Named entity recognition
  • Question answering (extractive)
Decoder-Only (GPT, Claude, Gemini, Llama)

Causal / Autoregressive — each token attends ONLY to past tokens

Training task: next-token prediction on enormous corpora

Best for:

  • Text generation, chat, coding
  • Open-ended question answering
  • AI agents (multi-step reasoning)
  • ALL modern frontier LLMs
Why are all modern LLMs decoder-only? Autoregressive generation is natural — generate one token at a time, using all previously generated tokens as context. The training task (predict next token) scales beautifully with data and compute, leading to emergent capabilities at large scale.
What is an LLM — Pretraining & Emergent Abilities CRITICAL

Pretraining

A very large transformer decoder trained on a single task: predict the next token given all previous tokens. Trained on trillions of tokens from the internet, books, code, and more.

From this one task, the model learns grammar, facts, reasoning patterns, multiple languages, coding syntax, and how to follow instructions — all as a side effect of being very good at next-token prediction.

Emergent Abilities

At certain model sizes, capabilities appear that were not present in smaller models: few-shot learning, chain-of-thought reasoning, arithmetic, code generation. These "emerge" — they are not explicitly trained for.

Training Pipeline for a Modern LLM
1. PretrainingNext-token prediction on ~1-10T tokens. Creates the "base model".
2. Supervised Fine-tuning (SFT)Train on (instruction, response) pairs. Model learns to follow instructions.
3. RLHFHuman raters compare responses → train a reward model → RL makes the LLM maximize reward. This makes the model "helpful, harmless, honest".
Base GPT-3 → SFT → RLHF = ChatGPT. This exact pipeline was described in the InstructGPT paper (2022) and is now used by everyone.
Tokenization & Byte Pair Encoding (BPE) HIGH

Why Not Characters?

Too many steps for long sequences. Model can't learn long-range patterns efficiently.

Why Not Words?

Vocabulary explosion (millions of words). Can't handle rare words, new words, or different languages.

BPE — The Sweet Spot

Start with individual characters. Iteratively merge the most frequently co-occurring pair into a new token. Repeat until vocabulary size is reached (typically 32K–100K tokens).

"tokenization" → ["token", "ization"] OR → ["tok", "en", "ization"] Depends on training corpus
Key Facts About Tokens
Rule of thumb1 token ≈ 0.75 words (English text)
GPT-4o vocabulary~100,000 tokens
Code is token-heavyIndentation, symbols use many tokens
Non-English textMore tokens per word than English
Why it mattersContext window limits, API cost, model performance all depend on token count
128K token context ≈ 96,000 words ≈ 380 pages. But just because something fits in context doesn't mean the model pays equal attention to all of it (lost-in-the-middle problem).
Text Generation: Temperature & Top-p Sampling HIGH
SettingBehaviorBest For
T = 0 (Greedy)Always pick highest probability token. Deterministic.Code generation, factual Q&A where consistency matters
T = 0.3–0.7Conservative sampling. High-prob tokens dominate. Coherent & focused.Customer support, factual summaries
T = 1.0Sample from natural model distribution. Balanced.General chat, default for most APIs
T = 1.5–2.0More random. Flatter distribution. Creative but may ramble.Creative writing, brainstorming, poetry
Top-p = 0.9Sample from smallest set of tokens accounting for 90% of probability. Ignores long tail.Most production systems — better than pure temperature
Interview answer: "Temperature scales the logits before softmax. At T=0 it's deterministic (greedy). At T=1 it samples from the natural model distribution. Higher T → more creative and random. Top-p (nucleus sampling) is more robust than temperature alone because it adapts to the shape of the distribution — if the model is very confident, fewer tokens are sampled from, regardless of temperature."
LoRA — Fine-tuning Without Full Retraining HIGH

The Problem

Full fine-tuning a 7B model: ~28GB GPU memory, hours of training, expensive.

LoRA Solution

Keep original weights frozen. Add small trainable low-rank matrices to each attention layer:

W_new = W_original + α × (A × B) A: (d × r), B: (r × d), rank r ≪ d

Only A and B are trained. r=8 or r=16 is typical.

Result: 99%+ reduction in trainable parameters. A 7B model can be fine-tuned on a single consumer GPU (≈6GB VRAM with QLoRA).

When to Fine-tune vs Prompt
Use PromptingTask works well in context. Need quick iteration. Don't want to maintain model versions.
Use Fine-tuningConsistent format needed. Proprietary style. Prompting reaches its limit. High volume (fine-tuned model cheaper per call).
SDE interview rule: "Prompt first, fine-tune only if prompting reaches its ceiling." Fine-tuning is expensive and creates maintenance burden.
Hallucination — The #1 LLM Question in SDE Interviews CRITICAL
What it is: The model generates confident, fluent, plausible-sounding text that is factually wrong.
Why it happens: LLMs are trained to produce coherent next tokens, not to verify facts. When the model doesn't know something, it extrapolates — and it doesn't have a reliable "I don't know" mechanism by default.

4 Mitigation Strategies

1. RAG (Best Fix in Production)

Ground responses in retrieved documents. System prompt: "Answer ONLY using the provided context." If the fact isn't in the context, the model can't make it up from memory.

2. Chain-of-Thought Prompting

"Let's think step by step" forces reasoning before answering. Each step can be checked. Significantly reduces hallucination on factual and reasoning tasks.

3. Explicit "I Don't Know"

System prompt: "If you're not certain, say 'I don't have information on that.' Never fabricate." This helps but doesn't eliminate hallucination.

4. Citation Verification

Ask the model to cite sources. Then verify those sources actually exist in retrieved chunks. If a cited source doesn't exist → flag the response.