AI System Design for SDEs — Prepflix AI Notes

The AI System Design Framework START HERE

Use this exact structure in any AI system design interview. It signals you think like an engineer, not just a researcher.

Clarify requirements — online vs. batch? latency SLA? scale? cost budget?
Define the ML task — what does the model predict/generate?
Data pipeline — collection, labeling, storage, freshness
Model choice — build vs. buy, fine-tune vs. prompt
Serving architecture — latency, throughput, scaling
Evaluation — offline metrics + online A/B test
Failure modes & safety — what can go wrong?
Monitoring — data drift, model drift, cost alerts

Key Questions to Ask Upfront

Latency requirement?<200ms (real-time) vs. <10s (acceptable)

QPS?1 req/s → single server; 10K/s → distributed

Freshness?Real-time data or batch OK?

Cost limit?GPT-4 at scale is expensive; consider smaller models

Domain-specific?General LLM or fine-tuned needed?

Design: RAG Pipeline at Scale MOST ASKED

1

Ingestion — Documents → Chunk → Embed → Store in vector DB. Use async workers (Celery/Cloud Tasks). Incremental indexing with document fingerprints to skip re-processing unchanged docs.

2

Retrieval — User query → Embed (same model as docs!) → ANN search → Top-K candidates. Add metadata filtering (date, source, category) to pre-filter before vector search.

3

Re-ranking — Cross-encoder re-scores top-K for precision. Cohere Rerank or bge-reranker. Expensive but worth it for high-stakes queries.

4

Generation — Inject top-N chunks into LLM prompt. Stream response to client. Set max_tokens to prevent runaway cost.

5

Caching — Semantic cache: embed the query → check if near-duplicate exists → return cached response. Saves 40–70% of LLM calls in practice. Use Redis or Momento.

Scale Bottlenecks

Component	Bottleneck	Solution
Embedding	GPU cost at scale	Batch embed, smaller model (e5-small)
Vector search	Latency at 100M+ vectors	HNSW with proper ef_search
LLM generation	Cost + latency	Semantic cache, smaller model for simple Qs
Context window	Too many chunks	Re-rank, limit to top-3

Numbers to Know

Embedding latency~20ms (e5-small, batch=1)

HNSW search~1–5ms for 1M vectors

GPT-4o generation~2–8s for 500 token response

Cross-encoder re-rank~50–200ms for top-20

Semantic cache hit rate40–70% in production

Design: Semantic Search System

Architecture

User Query ↓ Query Preprocessor (spell check, expand acronyms) ↓ Dual Retrieval: ├── BM25 (keyword, fast) ← Elasticsearch └── Dense (semantic embedding) ← Qdrant/Pinecone ↓ Reciprocal Rank Fusion (RRF) (merge BM25 + dense results) ↓ Cross-encoder Re-ranker ↓ Result Renderer + Explain

Hybrid Search = BM25 + Dense

Neither keyword nor semantic alone is best. Combine them:

RRF_score(d) = Σ 1/(k + rank_i(d))

k=60 is a common default. This is robust and doesn't require tuning per-query weights.

When to use what: BM25 wins for exact product codes, names, typos. Dense wins for synonyms, intent, paraphrasing.

Relevance Evaluation

nDCG@10Normalized discounted cumulative gain, gold standard

MRRMean reciprocal rank — first relevant result position

Recall@KDid relevant docs appear in top K?

Design: LLM-Powered API at Scale

Core Architecture

Client → API Gateway → Rate Limiter → Request Queue ↓ Semantic Cache (Redis) ↙ ↘ Cache Hit Cache Miss ↓ ↓ Return cached Prompt Builder ↓ LLM Router (cheap model if simple Q, powerful model if complex) ↓ LLM Provider API ↓ Response Validator ↓ Streaming to Client

LLM Router Pattern

Route queries to cheaper models when possible:

if query_complexity == "simple": model = "gpt-4o-mini" # $0.15/1M tokens elif requires_reasoning: model = "gpt-4o" # $2.50/1M tokens else: model = "claude-haiku" # cheapest

Complexity classifier can be a small fine-tuned model or rule-based (length, keywords).

Rate Limiting Strategy

Per-user TPM limitTrack token usage in Redis counter with TTL

Global QPM limitProtect against provider rate limits

Priority queuesPremium users skip the queue

Exponential backoff429 errors from provider → back off

Provider fallbackOpenAI down → Anthropic/Gemini

Model Serving — Self-Hosted vs. API

Approach	Latency	Cost (high QPS)	Privacy	When to Use
OpenAI/Anthropic API	2–10s	High	Data leaves org	Prototyping, low volume, general tasks
Self-hosted (vLLM)	0.5–3s	Low at scale	Full control	High QPS, sensitive data, cost optimization
Fine-tuned + served	0.2–1s	Lowest	Full control	Domain-specific, latency-critical
Quantized (4-bit)	Fast	Very low	Full control	Edge deployment, memory-constrained

vLLM Key Concepts

PagedAttentionKV cache in non-contiguous blocks; 2–4x throughput

Continuous batchingDynamic batching of in-flight requests

Speculative decodingSmall draft model → verify with big model → faster

Tensor parallelismSplit model across multiple GPUs

LoRA servingServe multiple fine-tuned adapters on 1 base model

QuantizationFP16 → INT8/INT4; 2–4x memory reduction

Evaluation Framework for AI Features

Offline Evaluation

Before shipping. Use golden test set (human-labeled Q&A pairs).

Metric	What It Measures
RAGAS Faithfulness	Answer grounded in retrieved context?
RAGAS Answer Relevance	Answer addresses the question?
BLEU / ROUGE	Text similarity (weak proxy)
LLM-as-judge	GPT-4 scores outputs 1–5
Human eval	Gold standard, slow

Online Evaluation (A/B Test)

Thumbs up/down rateUser explicit feedback

Session completion rateDid user get what they needed?

Fallback rate% of queries that triggered fallback

Retry rateUser rephrased = bad answer likely

Cost per queryToken usage × price

p95 latency95th percentile, not average

Interview Q: "How do you know your RAG system is getting better after each iteration?" — Answer: Maintain a golden eval set of 100–500 (question, expected_answer, source_doc) triples. After each change, run RAGAS (faithfulness + answer relevance) + LLM-as-judge. Track metrics in Weights & Biases or MLflow. Only ship if offline metrics improve without regressions. Then A/B test online.

Monitoring AI Systems in Production

What to Monitor	Signal	Action
Input data drift	Query embedding distribution shifts	Re-embed and re-index docs
Retrieval quality	Avg similarity score of retrieved chunks drops	Tune chunking or embedding model
LLM errors	Rate of 4xx/5xx from provider	Fallback provider, retry logic
Latency P95	Spikes > SLA	Check cache hit rate, queue depth
Cost per query	Token usage trending up	Shorten prompts, downgrade model
Hallucination rate	LLM-judge scores decline	Add retrieval grounding, stricter prompt
User thumbs down	Spike in negative feedback	Sample + human review bad examples

Tools: LangSmith (traces), Arize Phoenix (drift), Grafana (infra), Datadog (logs), Weights & Biases (experiments), Sentry (errors).