NLP & Transformers: Word Embeddings, BERT, GPT, RAG & AI Agents

Natural Language Processing (NLP) is the field that makes computers understand, interpret, and generate human language. For most of AI history, NLP was a graveyard of clever heuristics and hand-crafted rules. Then word embeddings came along, then RNNs, and then — in 2017 — a paper titled "Attention is All You Need" changed everything. Today, every major language model from ChatGPT to Gemini to Claude is built on the Transformer architecture described in that paper.

In this guide, we'll build your understanding from the ground up: starting with how raw text gets processed, building up through embeddings and sequence models, and finishing with the modern stack of Transformers, fine-tuning, RAG, and AI Agents. Every concept is connected to the next one.

Before any model sees text, that text goes through a processing pipeline. Think of it like a factory assembly line — raw material enters one end, and a structured representation comes out the other.

The five stages are: Raw Text (the original string) → Tokenization (split into tokens) → Embedding (convert tokens to vectors) → Model (process vectors through layers) → Output (prediction, generated text, or embeddings). Understanding each stage is crucial because bugs can creep in at any point.

Computers love numbers. Text, unfortunately, is not numbers. Worse, text is deeply ambiguous in ways that are effortless for humans but brutal for machines.

Other challenges include: morphology ("run", "running", "ran" are the same concept but different strings), syntax (word order changes meaning), coreference ("The trophy didn't fit in the suitcase because it was too big" — what is "it"?), and pragmatics (sarcasm, idioms, cultural context).

The naive approach: split on spaces. "I love NLP" becomes ["I", "love", "NLP"]. Works for simple cases but completely fails with "don't" (is it "don't" or "do" + "n't"?), punctuation, and non-English languages like Chinese which have no spaces at all.

BPE is the algorithm used by GPT-2, GPT-3, GPT-4, and many others. The idea is elegant: start with individual characters, then iteratively merge the most frequent pairs into new tokens. After enough merges, common words become single tokens while rare words get split into subword pieces.

Used by models like T5 and many multilingual models. Unlike BPE which requires pre-tokenized whitespace-split text, SentencePiece treats the input as a raw stream of characters including spaces (represented as a special character ▁). This makes it language-agnostic — it works for Japanese, Chinese, Arabic, and any other language without modification.

Instead of representing "king" as an integer ID, what if we represent it as a 300-dimensional vector of real numbers? The key insight is: words that appear in similar contexts should have similar vectors. This is the distributional hypothesis — "You shall know a word by the company it keeps."

Word2Vec achieves this by training a tiny neural network to predict: given the word "king", what words are likely to appear nearby? The hidden layer weights become the embeddings. The actual prediction task is discarded — we only keep the weights. This is called a self-supervised learning task: no human labels needed, just raw text.

CBOW (Continuous Bag of Words): Given context words ["The", "cat", "on", "the", "mat"], predict the center word "sat". Faster to train, works better for frequent words.

Skip-gram: Given the center word "sat", predict context words. Slower, but works better for rare words and produces better embeddings overall. Most implementations use Skip-gram with negative sampling (SGNS).

Word2Vec looks at local windows of text — it only sees words within a few positions of each target word. GloVe (Global Vectors for Word Representation, from Stanford) takes a different approach: build a giant co-occurrence matrix counting how often every word pair appears together across the entire corpus, then factorize this matrix into word vectors.

In practice, GloVe and Word2Vec produce similarly useful embeddings. GloVe tends to be slightly better at linear analogies because it explicitly optimizes for the ratio of co-occurrence probabilities.

Both Word2Vec and GloVe give every word exactly one vector, forever. This means "bank" always gets the same vector regardless of context. This is a fundamental limitation that contextual models like BERT were designed to fix.

An RNN reads text sequentially — one word at a time — while maintaining a "hidden state" that acts like short-term memory. Think of it like reading a book: as you read each word, your understanding updates. When you finish a sentence, your mental state captures its meaning.

At each time step t, the new hidden state h_t depends on the previous hidden state h_(t-1) and the current input x_t. This is the recurrence relation that gives RNNs their name. The final hidden state h_T theoretically summarizes the entire sequence.

Here's the critical failure of basic RNNs: during backpropagation, gradients must travel back through every time step. If the gradient gets multiplied by a number less than 1 at each step (which the tanh activation tends to cause), it shrinks exponentially. After 50 steps, the gradient is essentially zero — the model has completely forgotten the beginning of the sequence.

Think of it like a game of telephone: you whisper a message through 50 people. By the end, the original message is unrecognizable. RNNs have the same problem — they simply cannot learn long-range dependencies like "The keys that were on the table near the window in the living room of the old house [were/was] rusty" (which "were" or "was" — depends on "keys" 20 words back).

LSTMs (Hochreiter & Schmidhuber, 1997) solve the vanishing gradient problem with a brilliant architectural trick: a separate "cell state" that runs alongside the hidden state, with explicit mechanisms (gates) controlling what information flows through.

The cell state flows through with only linear interactions (addition and multiplication by gates), allowing gradients to flow much more easily over long distances. LSTMs can handle sequences of hundreds of tokens where vanilla RNNs fail completely.

Instead of forcing the model to compress an entire sentence into a single hidden state (the bottleneck problem), attention allows the model to look at ALL positions in the input simultaneously when producing each output token — and weight them by relevance.

Imagine you're translating "The animal didn't cross the street because it was too tired." When you write the French word for "it" (which must agree in gender with what "it" refers to), your brain immediately looks back and connects "it" to "animal". Attention gives models this same ability — a direct connection between any two positions.

In self-attention, every word asks: "How relevant is every other word to me?" Each input token is projected into three vectors:

• Query (Q): "What am I looking for?"
• Key (K): "What do I offer?"
• Value (V): "What is my actual content?"

For each query, we compute dot products with all keys (measuring relevance), scale by √d_k to prevent exploding gradients, apply softmax to get a probability distribution (attention weights), then take a weighted sum of the values. The result: each token gets a new representation that's a blend of all other tokens, weighted by how relevant they are.

Running attention once gives you one "perspective" on the relationships. Multi-head attention runs attention h times in parallel (typically h=8 or h=12), each with different learned projection matrices. This lets different heads capture different types of relationships simultaneously:

• Head 1 might focus on syntactic relationships (subject-verb agreement)
• Head 2 might track coreference (pronoun → antecedent)
• Head 3 might capture semantic similarity
• Head 4 might attend to positional proximity

The outputs of all heads are concatenated and projected back to the model dimension. You get the richness of h different analysis lenses for the price of one forward pass.

The Transformer, introduced in "Attention is All You Need" (Vaswani et al., 2017), replaces recurrence entirely with attention. The result: a model that can process all tokens in parallel (huge training speedup) and captures long-range dependencies directly (no vanishing gradient over distance).

Attention is inherently order-agnostic — "cat sat mat" and "mat sat cat" produce identical attention patterns. That's a problem since word order matters enormously in language. The solution: add positional information to the embeddings before feeding them to the Transformer.

The original paper uses sinusoidal functions at different frequencies — even positions and odd positions get different formulas. The result is a unique position vector for every position that smoothly interpolates and generalizes to sequences longer than those seen during training. Modern models often use learned positional embeddings or Rotary Position Embeddings (RoPE) instead.

Each Transformer layer has the same structure:

1. Multi-Head Self-Attention — each token attends to all other tokens
2. Add & Norm — residual connection (x + attention_output) followed by LayerNorm. Residual connections are critical for training deep networks — gradients flow directly back without degrading.
3. Feed-Forward Network — two linear layers with a ReLU/GELU in between. Applied independently to each position. This is where most of the "knowledge" is stored (the FFN layers are much larger than the attention layers).
4. Add & Norm again — another residual + LayerNorm

This layer is stacked N times (BERT-base: 12, GPT-3: 96, GPT-4: rumored 120). More layers = more capacity to learn complex patterns.

BERT (Google, 2018) uses only the Transformer encoder. During pretraining, it learns from two tasks:

• Masked Language Modeling (MLM): 15% of tokens are randomly masked. BERT must predict the masked tokens from context. Crucially, since it's an encoder, it can look at tokens on BOTH sides of the mask simultaneously — this is why it's "bidirectional".
• Next Sentence Prediction (NSP): Given two sentences, predict if the second follows the first. (Later research found NSP isn't very useful — RoBERTa drops it entirely.)

BERT produces rich contextual embeddings — "bank" in a financial sentence vs "bank" in a river sentence gets completely different vectors. This made BERT state-of-the-art on almost every NLP benchmark when it was released.

GPT (OpenAI) uses only the Transformer decoder (without the cross-attention to an encoder — it's a decoder-only model). During pretraining, it performs next token prediction: given tokens 1 through t, predict token t+1. The causal mask ensures it can only attend to previous tokens, making this autoregressive generation possible.

GPT-2 (2019) showed that a sufficiently large language model could generate coherent text so convincingly that OpenAI initially withheld the full model citing misuse concerns. GPT-3 (175B parameters, 2020) showed remarkable few-shot learning — giving it 3 examples is often enough to adapt to a new task without any weight updates. GPT-4 (2023) incorporated vision and significantly improved reasoning.

Hugging Face is the GitHub of AI models. They host over 500,000 models and provide the transformers library that makes fine-tuning state-of-the-art models accessible in less than 20 lines of code. The workflow is: load pretrained model → tokenize your dataset → fine-tune → save and push to Hub.

Large language models bake knowledge into their weights during pretraining. This creates two problems: (1) they can't know about events after their training cutoff, and (2) they hallucinate confidently — if they don't know something, they make it up rather than saying "I don't know".

RAG (Retrieval-Augmented Generation, Facebook/Meta 2020) solves this elegantly: instead of relying on memorized knowledge, retrieve relevant documents at query time and include them in the context. The model's job changes from "remember and answer" to "read and answer".

Offline (Indexing): Chunk your documents into pieces of ~256-512 tokens. Embed each chunk using an embedding model (e.g., text-embedding-3-small from OpenAI, or BAAI/bge-m3 open-source). Store the vectors in a vector database (Pinecone, Weaviate, pgvector, Chroma).

Online (Querying): Embed the user's query using the same embedding model. Run Approximate Nearest Neighbor (ANN) search to retrieve the top-K most similar chunks. Inject those chunks into the LLM prompt as context. The LLM reads the retrieved context and generates a grounded answer.

A single LLM call can answer questions, but it can't check a live stock price, run Python code, or book a flight. AI Agents solve this by giving the LLM access to tools and having it decide, step by step, which tool to use.

ReAct (Yao et al., 2022) interleaves reasoning traces and action calls. The agent generates a thought ("I need to check the weather in London"), then an action (call weather_api("London")), receives an observation ({"temp": "12°C", "weather": "cloudy"}), generates another thought, and continues until it has a final answer.

• In-context memory: The conversation history in the LLM's context window. Simple but limited by context length.
• External memory (vector store): Long-term storage of past interactions, retrieved by semantic similarity. The agent can "remember" conversations from weeks ago.
• Episodic memory: Structured logs of past actions and their outcomes. Helps agents avoid repeating mistakes.
• Tool state: State maintained in external tools (databases, files, APIs).

• Explain attention without equations first. Say: "Every token looks at every other token and decides how much to borrow from them." Then add the Q/K/V formulation.
• Know the BERT vs GPT distinction cold. Bidirectional vs causal, MLM vs next-token prediction, understanding vs generation — this comes up in almost every ML interview.
• Know why we need positional encoding — attention is order-agnostic and we must explicitly add position information.
• Be able to sketch the RAG pipeline — retrieval → augmentation → generation, with the offline indexing step.
• For fine-tuning questions: Always mention learning rate (2e-5), overfitting risk with small datasets, and when you'd use LoRA instead of full fine-tuning (discussed in Module 7).
• Vanishing gradient question: Explain both the RNN problem AND how Transformers avoid it (no sequential recurrence, residual connections).