How ChatGPT Works: A Plain-English Explanation for Software Engineers

Most explanations of ChatGPT are either "it's autocomplete" (true but useless) or a page of matrix equations (correct but impenetrable). As a software engineer you need something in between: a mental model precise enough to predict behaviour, debug prompts, estimate cost and latency, and design systems around the model. This guide follows a single message from your keyboard to the streamed reply, explaining each stage with small, runnable examples.

A caveat: OpenAI has not published full architecture and training details for its recent models. What follows describes the publicly documented approach (for example, the InstructGPT paper that preceded ChatGPT) and the general recipe shared across today's large language models. Specific numbers are illustrative.

The Big Picture in Five Steps

1. Tokenize. Your message, plus the system prompt and conversation history, is converted to a list of integer token IDs.
2. Embed. Each ID becomes a vector of numbers.
3. Transform. The vectors flow through dozens of transformer layers, where attention mixes information across tokens.
4. Predict. The final layer produces a score for every token in the vocabulary; softmax turns scores into probabilities.
5. Sample and repeat. One token is chosen, streamed to you, appended to the input, and the loop runs again until an end-of-message token appears.

Everything else, including training, safety, tools and memory, is about making step 4 produce good probabilities and about controlling what goes into step 1.

Step 1: Tokens, the Model's Alphabet

Models do not see characters or words. They see tokens from a fixed vocabulary, typically tens of thousands to a few hundred thousand entries. Most modern tokenizers use variants of byte-pair encoding (BPE): start from bytes and repeatedly merge the most frequent adjacent pair into a new token. Common words become single tokens; rare words split into pieces.

from collections import Counter

def train_bpe(words, num_merges):
    # Each word starts as a tuple of characters, with its corpus frequency
    vocab = Counter(tuple(w) for w in words)
    merges = []
    for _ in range(num_merges):
        pairs = Counter()
        for word, freq in vocab.items():
            for a, b in zip(word, word[1:]):
                pairs[(a, b)] += freq
        if not pairs:
            break
        best = max(pairs, key=pairs.get)
        merges.append(best)
        new_vocab = Counter()
        for word, freq in vocab.items():
            out, i = [], 0
            while i < len(word):
                if i + 1 < len(word) and (word[i], word[i + 1]) == best:
                    out.append(word[i] + word[i + 1]); i += 2
                else:
                    out.append(word[i]); i += 1
            new_vocab[tuple(out)] += freq
        vocab = new_vocab
    return merges, vocab

corpus = "low low low lower lowest newer newest new new".split()
merges, vocab = train_bpe(corpus, 4)
print(merges)              # [('l','o'), ('lo','w'), ('n','e'), ('ne','w')]
print(list(vocab))         # words now made of learned sub-word tokens

Why engineers should care about tokens:

• Cost and limits are in tokens. API pricing, rate limits and context windows all count tokens. In English, a token is roughly three to four characters on average.
• Languages differ. Text in many non-Latin scripts, including several Indian languages, has historically needed more tokens per word than English, making it slower and costlier. Newer tokenizers with larger vocabularies have narrowed the gap, but it is worth measuring for Hindi, Tamil or Bengali workloads.
• Letter-level tasks are awkward. The model sees "strawberry" as one or a few tokens, not ten letters, which is why counting letters or reversing strings was a famous weakness.
• Numbers tokenize unevenly, one reason plain arithmetic on long numbers is error-prone without tools.

Step 2: Embeddings, Turning IDs Into Meaning

Each token ID indexes a row in an embedding matrix, yielding a vector of a few thousand numbers. Training arranges these vectors so related tokens end up in related directions. Because a transformer otherwise has no sense of order, position information is added too; many modern models use rotary position embeddings (RoPE), which encode relative position by rotating query and key vectors inside attention.

These internal vectors are related to, but not the same as, the embeddings you get from an embeddings API for search; those come from models trained specifically to produce one vector per text for similarity. See embeddings and semantic search for that use.

Step 3: The Transformer and Attention

The transformer (introduced in the 2017 paper "Attention Is All You Need") is a stack of identical layers, often several dozen. Each layer has two parts, each wrapped with a residual connection and normalisation:

• Self-attention: each token gathers information from other tokens.
• Feed-forward network (MLP): each token's vector is transformed independently. Much of the model's stored knowledge is thought to live in these weights.

Attention, intuitively

Consider "The bank raised interest rates, so Priya moved her savings there." To interpret "there", the model must connect it to "bank", and to read "bank" as a financial institution it should notice "interest rates" and "savings". Attention does exactly this. Each token computes three vectors:

• a query ("what am I looking for?"),
• a key ("what do I contain?"),
• a value ("what will I pass on if selected?").

A token's query is compared (dot product) with every earlier token's key; the scores pass through softmax to become weights; the token's new representation is the weighted sum of the values. "there" ends up with a high weight on "bank".

Three details matter:

• Multi-head attention runs many attention operations in parallel per layer, each free to track different relationships (syntax, coreference, topic).
• Causal masking: a token can attend only to itself and earlier tokens, never to future ones. That is what makes the model a next-token predictor and what makes the KV cache possible.
• Cost: comparing every token with every earlier token is quadratic in sequence length, which is why long contexts are expensive. Serving systems cache keys and values from earlier tokens; our inference optimization guide covers that machinery and its memory cost.

Stacking layers lets representations become progressively more abstract: early layers handle local syntax, later layers capture meaning, task structure and what to say next. For the full mathematical treatment, see NLP and transformers.

Step 4: Picking the Next Token

After the last layer, the vector at the final position is multiplied by an output matrix to produce one score (logit) per vocabulary token. Softmax converts logits to probabilities. Then a sampling strategy picks one token:

import math, random

def softmax(logits, temperature=1.0):
    scaled = [l / temperature for l in logits]
    m = max(scaled)
    exps = [math.exp(s - m) for s in scaled]   # subtract max for stability
    total = sum(exps)
    return [e / total for e in exps]

def sample(tokens, logits, temperature=1.0, top_p=1.0, rng=random):
    probs = softmax(logits, temperature)
    ranked = sorted(zip(tokens, probs), key=lambda tp: tp[1], reverse=True)
    kept, cum = [], 0.0
    for tok, p in ranked:                     # nucleus: smallest set with mass >= top_p
        kept.append((tok, p)); cum += p
        if cum >= top_p:
            break
    z = sum(p for _, p in kept)
    r, acc = rng.random() * z, 0.0
    for tok, p in kept:
        acc += p
        if r <= acc:
            return tok
    return kept[-1][0]

tokens = ["Mumbai", "Delhi", "Pune", "banana"]
logits = [4.0, 3.0, 2.0, -1.0]                # "The capital of Maharashtra is ..."
for t in (0.2, 1.0, 2.0):
    print(t, [round(p, 3) for p in softmax(logits, t)])
rng = random.Random(7)
print([sample(tokens, logits, temperature=0.7, top_p=0.9, rng=rng) for _ in range(8)])

Running it shows the effect: at temperature 0.2 nearly all probability sits on "Mumbai"; at 2.0 the distribution flattens and even "banana" gets a few percent. Top-p (nucleus) sampling then trims the long tail of unlikely tokens. Key consequences:

• The same prompt can give different answers. That is sampling, not a bug. Low temperature makes outputs more repeatable but, on hosted APIs, still not always bit-identical because of batching and floating-point effects.
• Each token is committed once chosen. The model cannot go back and revise; an early wrong turn shapes everything after. Reasoning models mitigate this by producing a long internal reasoning sequence first, where they can notice and correct mistakes before the final answer.
• Streaming works naturally: each token is sent as soon as it is sampled.

Pretraining: Learning From a Vast Amount of Text

A freshly initialised model outputs noise. Pretraining fixes that with one deceptively simple objective: for every position in a huge corpus of text (web pages, books, code, papers and more, filtered and deduplicated), predict the next token, and adjust the weights by gradient descent to make the correct token more likely. The loss is cross-entropy: the negative log probability assigned to the actual next token.

Why does such a simple objective produce something that seems to understand? Because predicting text well requires modelling what text is about. To predict the next token of a Python function, it helps to model what the code does. To predict the end of "The capital of Maharashtra is", it helps to store the fact. To predict the next step of a worked solution, it helps to model the reasoning. Scale (more parameters, more data, more compute) has reliably improved this, as studied in scaling-law research.

The result is a base model: an extraordinarily capable text continuer, but not an assistant. Ask a base model "What is the capital of India?" and it may continue with three more quiz questions, because that is what such text often looks like on the web. Pretraining is also where the knowledge cutoff comes from: the model knows only what was in its training data.

Post-Training: SFT, RLHF and Reasoning

Supervised fine-tuning (SFT)

The base model is fine-tuned on curated examples of conversations in which an ideal assistant responds to prompts: answering questions, following formatting requests, writing code, declining harmful requests. This teaches the format and style of being an assistant.

Reinforcement learning from human feedback (RLHF)

Writing perfect demonstrations is expensive, but comparing two responses is easy. RLHF, as described in OpenAI's InstructGPT work, uses that:

1. Sample several responses to each prompt from the model.
2. Human labellers rank them by helpfulness, honesty and harmlessness.
3. Train a reward model to predict those rankings.
4. Optimise the language model with reinforcement learning (originally PPO) to produce responses the reward model scores highly, with a penalty for drifting too far from the SFT model.

Variants are now common: DPO (direct preference optimisation) learns from preference pairs without a separate reward model and RL loop, and approaches such as Anthropic's Constitutional AI use AI feedback guided by written principles. Preference training is powerful but has side effects, notably sycophancy: models learn that agreeable, confident answers are rated well, and labs actively work to counter this.

Reasoning models

Since 2024, a major advance has been training models with reinforcement learning on problems whose answers can be checked automatically, such as maths with known answers and code with tests. The model learns to generate a long chain of intermediate reasoning before answering, and is rewarded when the final answer is correct. OpenAI's o1 series and DeepSeek-R1 were prominent early examples, and "thinking" modes are now standard across major model families. These models spend more tokens (and time and money) per answer in exchange for much better performance on multi-step problems; many products let you choose how much reasoning effort to spend.

The Chat Application Around the Model

"ChatGPT" is a product: a model plus a lot of application engineering. Understanding the split explains many behaviours.

Roles and chat templates

Conversations are serialised into a single token sequence with special tokens marking roles, conceptually:

<|system|> You are a helpful assistant. Today's date is ...
<|user|> Explain recursion to a 10-year-old.
<|assistant|> Imagine you have a set of Russian dolls ...
<|user|> Now give a Python example.
<|assistant|>

The model continues from the final assistant marker. Post-training teaches it to give system instructions priority and to stop at the end of its turn. (Exact formats differ between models and are handled for you by APIs.)

Statelessness and memory

The model has no memory between calls. Every turn, the application sends the relevant history again. Cross-conversation "memory" features work by the application saving notes and inserting them into future contexts. Nothing you type changes the model's weights in real time.

Tools

The model can be trained to emit structured tool calls: "search the web for X", "run this Python", "generate an image". The application executes the tool and inserts the result into the context, and the model continues. This is how ChatGPT can cite current news or compute exact numbers: not by knowing them, but by fetching them. The same mechanism underlies AI agents and retrieval-augmented generation (RAG).

Safety layers

Beyond training, products typically add moderation classifiers on inputs and outputs, system prompts with policies, and monitoring for abuse.

Context Windows: The Model's Working Memory

The context window is the maximum number of tokens (input plus output) the model can process at once. Modern frontier models support context windows from around a hundred thousand to over a million tokens, but bigger is not free:

• Cost and latency grow with context. You pay for every input token on every turn, and long prompts increase time to first token. That is why chat apps summarise or trim old history, and why prompt caching discounts matter.
• Attention to long contexts is uneven. Research such as "Lost in the Middle" found models used information at the beginning and end of long contexts better than in the middle. Newer models are much better, but retrieval quality still degrades with very long, noisy inputs.
• Everything in context is influence. Retrieved documents and tool outputs can contain instructions; this is the root of prompt injection.

Limitations and Why They Happen

Practical Takeaways for Software Engineers

• Prompts are programs over a probability distribution. Clear instructions, examples and structure shift probabilities towards what you want. See prompt engineering for developers.
• Give the model the facts rather than hoping it knows them: retrieval, tools and explicit context beat memorised knowledge for anything that matters.
• Budget in tokens. Estimate input and output tokens per request to predict cost and latency.
• Constrain outputs with structured output modes and validate them like any untrusted input.
• Measure with evaluation sets rather than eyeballing a few responses.
• Use reasoning models selectively: they cost more tokens and time but shine on multi-step problems.

How This Comes Up in Interviews

"Explain how ChatGPT or an LLM works" is now a common question in AI engineer interviews, ML rounds and even general SDE interviews at product companies. A strong answer is layered: a one-minute version, then depth wherever the interviewer probes. Practise these:

• "Walk me through what happens when I send a message." Tokenize, embed, transformer layers with causal attention, logits, softmax, sampling, loop; app adds history and tools.
• "What is attention and why does it matter?" Query-key-value, weighted mixing of earlier tokens, multi-head, quadratic cost.
• "Difference between pretraining, fine-tuning and RLHF?" Next-token prediction on vast text; supervised demonstrations; preference optimisation with a reward model.
• "Why do LLMs hallucinate and how would you reduce it in a product?" Plausibility objective; RAG, tools, citations, abstention and evals.
• "What does temperature do?" Rescales logits before softmax; trade-off between determinism and diversity.

If you can explain each of these to a non-ML engineer in plain language and then go one level deeper when asked, you understand LLMs better than most people who use them daily. From here, the natural next steps are the generative AI and LLMs deep dive and building a small RAG application end to end.

Frequently Asked Questions

Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.