Design an API Gateway: The Complete System Design Guide

"Design an API gateway" shows up both as a full system design question and as a follow-up inside other designs ("you drew a gateway box, what does it actually do?"). Interviewers use it to test whether you understand the edge of a microservice architecture: where trust is established, where abuse is stopped, how a request is traced across twenty services, and what happens when one of those services is slow.

The answer below is structured around the life of a single request through the gateway, which is the most natural way to explain it on a whiteboard.

What an API Gateway Does (and What It Is Not)

Before designing, separate three things candidates often blur together.

Real systems layer them: DNS → CDN/WAF → L4 load balancer → gateway fleet → services (optionally through a mesh). A gateway is a reverse proxy with opinions.

Requirements

Functional

• Route requests to upstream services by host, path, method and headers (including canary/version routing).
• Terminate TLS and support HTTP/1.1, HTTP/2, gRPC and WebSocket upgrades.
• Authenticate callers: user JWTs, OAuth access tokens, API keys for partners, mTLS for B2B.
• Rate limit and quota per user, API key, IP and route.
• Transform requests/responses: header injection, path rewrite, protocol translation (REST to gRPC).
• Aggregate calls for some endpoints (or delegate to BFFs).
• Emit logs, metrics and traces for every request.
• Change configuration (new routes, plugins, limits) without restarts or dropped connections.

Non-functional

• Low added latency: a few milliseconds at p99 on top of the upstream call.
• Very high availability: if the gateway is down, everything is down. Target at least the availability of your best service.
• Horizontal scalability to hundreds of thousands of requests per second.
• Isolation: a slow upstream or a noisy client must not degrade unrelated routes.
• Secure by default: deny unknown routes, strip internal headers, validate input sizes.

Capacity Estimates

Assume a consumer app on the scale of a large Indian e-commerce or food delivery platform (illustrative numbers):

• 30M DAU × ~100 API calls/day = 3B requests/day ≈ 35k RPS average, ~150k RPS at peak (sale start, dinner rush).
• If one gateway instance handles ~10k RPS with TLS, JWT checks and a few plugins (a conservative figure; real numbers depend on payload sizes, plugins and hardware), peak needs ~15 instances. Run 30+ across three zones for N+1 zone failure and headroom.
• Connections: mobile clients keep connections alive; with HTTP/2 multiplexing, a few million concurrent client connections spread across the fleet is plausible, so tune file descriptors and memory per connection.
• TLS: full handshakes are CPU-expensive relative to resumed sessions. Session resumption and long-lived connections matter more than raw cipher speed.
• Logs: 3B requests × ~1 KB access log = 3 TB/day. Sample successful requests, keep all errors.

The gateway's own state is tiny (routes, keys, limits). The heavy state lives in the rate-limit store and the identity provider, which is why those get their own deep dives.

High-Level Architecture: Data Plane and Control Plane

The data plane is the fleet of proxies on the request path. It must be fast, stateless and boring. The control plane stores desired configuration, validates it, and distributes it. Envoy formalised this split with its xDS discovery APIs; Kong supports a similar hybrid mode where control-plane nodes push configuration to data-plane nodes.

The key property: data planes keep running on their last known good config if the control plane is unavailable. A control-plane outage should block config changes, never traffic.

The Request Pipeline (Filter Chain)

Every gateway, whatever the product, is an ordered chain of filters/plugins. Getting the order right is a real design decision.

1. Connection and TLS. Terminate TLS, negotiate HTTP/2, enforce max header and body sizes.
2. Request ID and trace context. Generate an id if absent; start or continue a W3C traceparent.
3. Sanitise. Strip headers that only the gateway may set (X-User-Id, X-Internal-*). Otherwise a client simply sends X-User-Id: 1 and becomes user 1.
4. Coarse protection. IP reputation, bot/WAF rules, per-IP rate limit. Cheap checks first.
5. Route match. Find the route; unknown routes return 404 here without touching services.
6. Authenticate. Validate JWT/API key/mTLS as the route requires.
7. Authorise (coarse). Scopes or roles required by the route, e.g. orders:write.
8. Rate limit and quota per identity and route.
9. Transform. Inject identity headers, rewrite paths, translate protocols.
10. Proxy. Pick an upstream instance (service discovery, load balancing), apply timeout, retry and circuit-breaker policy.
11. Response filters. Strip internal headers, compress, add CORS/security headers, record metrics and access log.

Here is a minimal, synchronous Java version of the pattern. Production gateways are asynchronous and non-blocking (Netty, Envoy's event loop), but the composition idea is the same.

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.function.Function;

record Request(String method, String path, Map<String, String> headers) {
    Request withHeader(String k, String v) {
        Map<String, String> h = new HashMap<>(headers);
        h.put(k, v);
        return new Request(method, path, h);
    }
    Request withoutHeader(String k) {
        Map<String, String> h = new HashMap<>(headers);
        h.remove(k);
        return new Request(method, path, h);
    }
}

record Response(int status, String body) {}

interface Filter {
    Response apply(Request req, Chain chain);
}

final class Chain {
    private final List<Filter> filters;
    private final Function<Request, Response> upstream;
    private final int index;

    Chain(List<Filter> filters, Function<Request, Response> upstream, int index) {
        this.filters = filters;
        this.upstream = upstream;
        this.index = index;
    }

    Response next(Request req) {
        if (index == filters.size()) return upstream.apply(req);
        return filters.get(index).apply(req, new Chain(filters, upstream, index + 1));
    }
}

final class RequestIdFilter implements Filter {
    public Response apply(Request req, Chain chain) {
        String id = req.headers().getOrDefault("X-Request-Id", UUID.randomUUID().toString());
        return chain.next(req.withHeader("X-Request-Id", id));
    }
}

final class AuthFilter implements Filter {
    interface TokenVerifier { String userIdOrNull(String bearerToken); }
    private final TokenVerifier verifier;
    AuthFilter(TokenVerifier verifier) { this.verifier = verifier; }

    public Response apply(Request req, Chain chain) {
        Request clean = req.withoutHeader("X-User-Id");           // never trust the client's copy
        String auth = clean.headers().getOrDefault("Authorization", "");
        if (!auth.startsWith("Bearer ")) return new Response(401, "missing token");
        String userId = verifier.userIdOrNull(auth.substring(7));
        if (userId == null) return new Response(401, "invalid token");
        return chain.next(clean.withHeader("X-User-Id", userId));
    }
}

final class RateLimitFilter implements Filter {
    interface Limiter { boolean tryAcquire(String key); }
    private final Limiter limiter;
    RateLimitFilter(Limiter limiter) { this.limiter = limiter; }

    public Response apply(Request req, Chain chain) {
        String key = req.headers().getOrDefault("X-User-Id", "anon") + ":" + req.path();
        return limiter.tryAcquire(key) ? chain.next(req) : new Response(429, "slow down");
    }
}

Each filter can short-circuit (return 401/429) or call chain.next. Plugins in Kong, filters in Envoy and GatewayFilters in Spring Cloud Gateway are all variations of this shape.

Routing: Matching Requests to Upstreams

A gateway might hold thousands of routes. A linear scan of regexes per request is slow and makes precedence confusing. Most gateways compile routes into a prefix tree (trie) or radix tree on path segments with clear precedence rules: exact segment beats parameter, parameter beats catch-all.

class Node:
    __slots__ = ("literal", "param", "param_name", "wildcard", "route")
    def __init__(self):
        self.literal, self.param, self.param_name = {}, None, None
        self.wildcard, self.route = None, None

class Router:
    def __init__(self):
        self.root = Node()

    def add(self, method, pattern, upstream):
        node = self.root
        for seg in [s for s in pattern.split("/") if s]:
            if seg == "**":                       # catch-all: must be last
                node.wildcard = node.wildcard or {}
                node.wildcard[method] = upstream
                return
            if seg.startswith("{") and seg.endswith("}"):
                if node.param is None:
                    node.param, node.param_name = Node(), seg[1:-1]
                node = node.param
            else:
                node = node.literal.setdefault(seg, Node())
        node.route = node.route or {}
        node.route[method] = upstream

    def match(self, method, path):
        segs = [s for s in path.split("/") if s]
        params = {}
        def walk(node, i):
            if i == len(segs):
                if node.route and method in node.route:
                    return node.route[method]
            else:
                child = node.literal.get(segs[i])          # 1. exact segment wins
                if child and (hit := walk(child, i + 1)):
                    return hit
                if node.param:                            # 2. then {param}
                    params[node.param_name] = segs[i]
                    if hit := walk(node.param, i + 1):
                        return hit
                    del params[node.param_name]
            if node.wildcard and method in node.wildcard:   # 3. then /**
                return node.wildcard[method]
            return None
        upstream = walk(self.root, 0)
        return (upstream, params) if upstream else None

r = Router()
r.add("GET", "/v1/users/{id}", "user-svc")
r.add("GET", "/v1/users/me", "profile-svc")
r.add("GET", "/v1/orders/**", "order-svc")
print(r.match("GET", "/v1/users/me"))        # ('profile-svc', {})
print(r.match("GET", "/v1/users/42"))        # ('user-svc', {'id': '42'})
print(r.match("GET", "/v1/orders/9/items"))  # ('order-svc', {})
print(r.match("POST", "/v1/users/42"))       # None

Matching cost is proportional to the number of path segments, not the number of routes (backtracking happens only when an exact branch dead-ends). Real gateways also match on host, headers and query parameters, and compile the table once per config version so the hot path never rebuilds it.

Advanced routing patterns

• Canary and weighted routing: send 5% of /v1/checkout to checkout-v2, by random weight or by hashing user id so each user sticks to one version.
• Header-based routing: X-App-Version < 8.2 goes to a legacy upstream.
• Versioning: /v1 and /v2 as separate routes; retire old versions by watching per-route traffic.
• Service discovery: upstreams resolve to healthy instances through Kubernetes endpoints, Consul or DNS, refreshed continuously, with active health checks and passive outlier detection.

Authentication at the Edge

Centralising authentication in the gateway means every service does not reimplement token validation, and unauthenticated junk never reaches your services. Background on token formats is in OAuth 2.0, OIDC and JWT explained.

Design points interviewers probe

• Key rotation: cache the JWKS, honour its cache headers, and refetch when a token arrives with an unknown kid (rate-limited so attackers cannot force refetch storms).
• Revocation: JWTs are valid until expiry. Keep access tokens short-lived (minutes) and, for urgent cases, push a small deny-list of revoked token ids or user ids to gateways. The full session design is in authentication and SSO design.
• Identity propagation: after validation, forward X-User-Id, scopes, and tenant as headers, or a short-lived internal token signed by the gateway. Services trust these only because the network path guarantees they came from the gateway (mTLS or network policy).
• Authorisation split: the gateway does coarse checks (is this token allowed to call /admin/*?). Fine-grained checks (can user 42 read order 9?) stay in the owning service, which knows who owns order 9.
• Store hashed API keys, never plaintext, and show the key to the partner only once.

Rate Limiting in a Gateway Fleet

Algorithms (token bucket, sliding window) are covered in depth in design a rate limiter. The gateway-specific question is where the counters live when you have 30 instances.

Decide the failure mode explicitly: if Redis is unreachable, fail open (allow traffic, rely on local caps) for user-facing APIs, but perhaps fail closed for expensive partner APIs. Return 429 with Retry-After and limit headers so well-behaved clients back off.

Request Aggregation and the BFF Pattern

A mobile home screen might need user profile, cart count, banners, recommendations and active orders. Five round-trips over a 4G connection in a Tier-2 city is painful. Aggregation solves it: the client makes one call, something server-side fans out in parallel.

import asyncio

async def call(name, coro, timeout, fallback):
    try:
        return name, await asyncio.wait_for(coro, timeout)
    except Exception:
        return name, fallback          # degrade this widget, not the whole screen

async def home_screen(user_id, svc):
    results = await asyncio.gather(
        call("profile", svc.profile(user_id), 0.15, None),
        call("cart", svc.cart_count(user_id), 0.10, 0),
        call("banners", svc.banners(user_id), 0.20, []),
        call("recs", svc.recommendations(user_id), 0.25, []),
    )
    data = dict(results)
    if data["profile"] is None:        # profile is mandatory for this screen
        raise RuntimeError("profile unavailable")
    return data

Note the per-call timeouts and fallbacks: the response time is bounded by the slowest required call, and optional widgets degrade independently.

Where should aggregation live?

Putting aggregation for every screen into the shared gateway is a classic anti-pattern: the gateway team becomes a bottleneck for every app release, and the gateway accumulates business logic. The Backend for Frontend (BFF) pattern, popularised by Sam Newman and SoundCloud, puts a thin backend per client experience (Android BFF, web BFF, partner API) owned by the client team.

The common shape at scale is: gateway for policy (auth, limits, routing) → BFFs for composition → domain services. The gateway stays generic; BFFs stay close to the UI.

Resilience: Timeouts, Retries, Circuit Breakers

The gateway sees every upstream failure first, so it is the right place for a consistent resilience policy.

• Timeouts per route, set from the upstream's real latency distribution (for example p99 × 2), never infinite. Total timeout for retries included.
• Retries only for idempotent methods (GET, PUT, DELETE, or POST with an idempotency key), only on retryable errors (connect failure, 503), with jittered backoff, and with a retry budget (retries capped at, say, 10% of requests) so a struggling service is not hit by a retry storm.
• Circuit breakers / outlier ejection: stop sending to an instance with consecutive 5xx; cap concurrent requests and pending requests per upstream so one slow service cannot consume all gateway connections (bulkheading).
• Load shedding: under overload, reject low-priority traffic (analytics beacons, prefetch) before checkout or payment.
• Graceful draining on deploy: stop accepting new connections, finish in-flight requests, then exit.

Plugins, Extensibility and Configuration

Plugin models

Configuration management

• Declarative and versioned: routes and plugins live in Git (or a DB with history); every change is reviewed and has an id.
• Validated before push: schema checks, conflict detection (two routes with identical match), dry-run against recorded traffic.
• Atomic, incremental push: data planes receive a full snapshot or deltas with a version; they apply it atomically and report ACK/NACK.
• Staged rollout: push to one zone or a few nodes first, watch error rates, then everywhere. Bad config is a leading cause of large outages, so treat config like code.

Observability: The Gateway as Your Best Sensor

Because every request passes through it, the gateway gives you uniform telemetry for free. Details on the three pillars are in observability: logs, metrics and tracing.

• RED metrics per route and upstream: rate, errors (split 4xx vs 5xx, and gateway-generated vs upstream-generated), duration histograms.
• Trace context propagation: create or continue traceparent, so a trace starts at the edge and follows the request through services.
• Structured access logs with request id, route id, upstream, latency breakdown (gateway time vs upstream time), identity (hashed), status. Sample 2xx, keep all errors.
• Distinguish the gateway's own latency from upstream latency, so "the API is slow" can be attributed in seconds.
• Per-consumer usage for API products: monthly calls per API key feed billing and quota dashboards.

Kong vs Envoy vs Managed Gateways

You will often be asked "would you build this or use something?" The honest answer is that almost nobody builds a gateway proxy from scratch; you pick one and build the control plane, policies and tooling around it.

A reasonable interview position: for a Kubernetes-heavy company already using Istio, an Envoy-based gateway keeps one proxy technology across edge and mesh. For a company selling APIs to partners, Kong or a managed API management product brings key management and developer portals. For a small team on AWS with spiky traffic, a managed gateway avoids running infrastructure, until per-request cost or limits push you off it. Always check current product documentation and quotas; these products evolve quickly.

The Interview Angle

Follow-up questions and crisp answers

• "Isn't the gateway a single point of failure?" Stateless fleet across zones behind L4 load balancers, last-known-good config, no synchronous dependency that can take it down (rate limiter and IdP failures have defined fail-open/closed behaviour).
• "How do you add a new route without a restart?" Control plane validates, versions and pushes config; proxies hot-swap the compiled route table atomically.
• "A downstream service is slow. What happens at the gateway?" Route timeout fires, retry budget prevents amplification, circuit breaker and concurrency limits isolate that upstream, other routes unaffected.
• "How do services know who the user is?" Gateway validates the token, strips client-provided identity headers, injects trusted ones; mTLS between gateway and services.
• "WebSockets through the gateway?" Supported as HTTP upgrade; authenticate at upgrade time, use connection-level limits, and remember long-lived connections complicate draining during deploys.
• "How does this relate to microservices?" See the broader context in microservices interview questions and microservices vs monolith.

Common mistakes

• Trusting client-supplied identity headers.
• Calling the identity provider on every request instead of validating JWTs locally.
• Retrying non-idempotent POSTs, or retrying without a budget.
• No per-route timeouts, so one slow service exhausts gateway connections.
• Forgetting the control plane: "we'll just edit the Nginx config and reload" does not scale to hundreds of routes and teams.
• Logging full request bodies (tokens, PII, card data) in access logs.

Frequently Asked Questions

Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.