Design a Payment System: How UPI & Payment Gateways Actually Work

Payment system design is the question where interviewers stop caring about QPS and start caring about correctness. A feed that shows a stale post is fine. A payment that debits a customer twice, or marks a debited payment as failed so the merchant never ships, is a support ticket, a regulatory complaint and a loss. This guide designs a payment platform the way a PSP or a gateway like Razorpay, PhonePe or Paytm would think about it: start from how UPI actually moves money, then build the services, data model and failure handling around it.

We will scope the interview version as: "Design a payment system that lets merchants accept UPI and card payments, with refunds, webhooks and settlements." That covers both the consumer-app side (you are the PSP) and the merchant side (you are the gateway). If your interviewer narrows it to a wallet, the digital wallet design is a closely related problem with a closed-loop ledger.

How a UPI Payment Flows: PSP, NPCI and Banks

Before designing anything, get the actors right. Interviewers in India frequently ask "walk me through what happens when I pay ₹500 on PhonePe", and a vague answer ("the app calls the bank") signals you have never thought about it.

The push payment ("Pay") sequence

1. Initiate. You scan a QR containing upi://pay?pa=shop@okicici&am=500&tr=ORD123. The app creates a transaction ID and shows the amount. You enter your PIN into NPCI's Common Library on the device, which encrypts it so neither the app nor the PSP sees it in clear text.
2. Payer PSP → NPCI. The app sends the request to its PSP backend, which forwards a pay request to the NPCI switch.
3. Address resolution. NPCI asks the payee's PSP to resolve shop@okicici to an account and IFSC (and gets the payee name, which is why you see "Paying Sharma General Store").
4. Debit. NPCI sends a debit request with the encrypted PIN to the remitter bank. The bank decrypts and verifies the PIN, checks balance and limits, debits the account, and replies with success and a reference.
5. Credit. NPCI sends a credit request to the beneficiary bank, which credits the payee and replies.
6. Response. NPCI sends the final response to both PSPs. The payer app shows the green tick; the merchant's soundbox announces "₹500 received".
7. Settlement. No money physically moves between banks at that moment. NPCI computes net positions between banks across settlement cycles during the day, and the actual interbank settlement happens through the banks' accounts at RBI.

A collect request reverses the initiation: the payee asks NPCI to send a request to the payer's app, the payer approves with a PIN, and the same debit/credit pipeline runs. UPI AutoPay adds a mandate created once with a PIN and executed later without one, within the limits that mandate allows.

For scale context, NPCI publishes monthly UPI volumes on its website; in 2025 they were on the order of 20 billion transactions a month. A single large PSP or gateway handles a large fraction of that, with sharp peaks during festival sales and month-start salary days.

Step 1: Clarify Requirements

Functional requirements

• Merchants create an order and accept payment via UPI (intent, QR, collect) and cards.
• Track each payment to a final state; expose status to merchant and customer.
• Full and partial refunds.
• Notify merchants via webhooks; let them poll as a fallback.
• Settle collected funds to merchant bank accounts (T+1 or similar), minus fees.
• Reconcile internal records with banks, NPCI and card networks daily.

Non-functional requirements

• Correctness over availability for money movement: no double charge, no lost payment, every rupee accounted for.
• High availability for accepting payments (an outage at 8 PM during a sale costs merchants real money). Target 99.99% on the payment API.
• Latency: our own overhead should be small (tens of milliseconds); total UPI latency is dominated by banks and is typically a few seconds.
• Auditability: immutable history, who changed what and when. Regulatory requirements in India include storing payment data in India.
• Security: PCI DSS for card data (tokenise, never store raw PANs outside a vault), encryption, HMAC-signed webhooks.

Say explicitly what is out of scope: fraud models, KYC onboarding, merchant dashboard UI, and building the card network itself.

Step 2: Capacity Estimates

Assume a large gateway: 100 million payments per day, peak at 5x average.

Takeaways: write volume is moderate but every write is transactional, so one Postgres or MySQL primary is not enough at peak; we will shard by merchant_id. Reads are cache-friendly for terminal states only. Storage grows tens of terabytes a year, so hot/cold tiering matters, but payment records must be retained for years for audit. If you want a refresher on this style of math, see back-of-the-envelope estimation.

Step 3: API Design

POST /v1/orders
  Idempotency-Key: 7f1c2a...           # generated by merchant server
  { "amount": 50000, "currency": "INR", "receipt": "ORD123" }
  → 201 { "order_id": "order_9A3", "status": "created" }

POST /v1/payments
  Idempotency-Key: c09e...
  { "order_id": "order_9A3", "method": "upi",
    "upi": { "flow": "intent" } }
  → 201 { "payment_id": "pay_77X", "status": "pending",
          "intent_url": "upi://pay?pa=...&tr=pay_77X&am=500.00" }

GET  /v1/payments/{payment_id}
  → 200 { "status": "captured", "amount": 50000, "rrn": "4281..." }

POST /v1/payments/{payment_id}/refunds
  Idempotency-Key: 11ab...
  { "amount": 20000, "reason": "partial return" }
  → 201 { "refund_id": "rfnd_12", "status": "pending" }

Webhook → merchant URL
  X-Signature: hex(HMAC_SHA256(secret, raw_body))
  { "event_id": "evt_5Q", "type": "payment.captured", "payload": {...} }

Design choices worth saying aloud: amounts are integers in paise, never floats. Every mutating endpoint takes an Idempotency-Key. Order and payment are separate because one order may see several payment attempts (UPI fails, customer retries with a card), and only one of them may succeed. The API returns pending immediately; the final state arrives asynchronously.

Step 4: Data Model

CREATE TABLE payments (
  payment_id      TEXT PRIMARY KEY,
  merchant_id     TEXT NOT NULL,
  order_id        TEXT NOT NULL,
  amount_paise    BIGINT NOT NULL CHECK (amount_paise > 0),
  method          TEXT NOT NULL,             -- upi | card | netbanking
  status          TEXT NOT NULL,             -- created|pending|authorized|captured|failed|refunded
  provider        TEXT,                      -- which bank/PSP route we used
  provider_ref    TEXT,                      -- RRN / UPI txn id / auth code
  version         INT NOT NULL DEFAULT 0,    -- optimistic locking
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX one_success_per_order
  ON payments(order_id) WHERE status IN ('captured','authorized');

CREATE TABLE payment_events (               -- append-only history
  event_id     BIGSERIAL PRIMARY KEY,
  payment_id   TEXT NOT NULL,
  from_status  TEXT, to_status TEXT NOT NULL,
  source       TEXT NOT NULL,               -- api|bank_callback|poller|recon
  raw_response JSONB,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE idempotency_keys (
  merchant_id     TEXT NOT NULL,
  idem_key        TEXT NOT NULL,
  request_hash    TEXT NOT NULL,
  response_code   INT,
  response_body   JSONB,
  locked_at       TIMESTAMPTZ,
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (merchant_id, idem_key)
);

CREATE TABLE ledger_entries (
  entry_id     BIGSERIAL PRIMARY KEY,
  txn_id       TEXT NOT NULL,               -- groups entries that must balance
  account_id   TEXT NOT NULL,               -- merchant_payable:m_42, upi_receivable:icici ...
  direction    CHAR(1) NOT NULL CHECK (direction IN ('D','C')),
  amount_paise BIGINT NOT NULL CHECK (amount_paise > 0),
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

Two details interviewers love. The partial unique index on order_id guarantees at the database level that an order can have at most one successful payment, even if two attempts race. And the payment_events table is append-only: the payments row is a convenient current-state projection, but the event history is what you show an auditor.

Payment IDs should be globally unique, roughly time-ordered and not guessable; a Snowflake-style ID with a random suffix works well (see unique ID generator design). Shard payments, idempotency_keys and payment_events by merchant_id so a merchant's writes and idempotency checks stay on one shard; very large merchants can be split further by hashing payment_id within a merchant group.

Step 5: High-Level Architecture

• Payment Service owns the payment lifecycle. It is the only writer of payment status.
• Router / Connectors wrap each external provider (one connector per PSP bank or acquirer). The router picks a route based on method, success rates and cost, and can fail over when one bank is degraded.
• Outbox + Kafka: state changes are written to the payments DB and an outbox table in the same transaction; a relay publishes them to Kafka. Ledger, webhooks, analytics and notifications consume from there. This avoids the dual-write bug where the DB commit succeeds but the event publish fails. More on the log in designing Kafka.
• Status Poller chases pending payments with the provider's status-check API.
• Reconciliation and Settlement run as batch plus streaming jobs against external files.

Deep Dive 1: The Payment State Machine

Write the state machine on the whiteboard. It is the single most useful artifact in this interview.

For UPI, authorization and capture happen together, so a successful UPI payment goes straight to CAPTURED. For cards, the authorization holds funds and capture can come later (hotels, marketplaces that capture on shipment).

Enforcing legal transitions

Transitions are checked in code and enforced with a conditional update, so two concurrent writers (a bank callback and the poller) cannot both apply a transition:

UPDATE payments
   SET status = 'captured', provider_ref = $2,
       version = version + 1, updated_at = now()
 WHERE payment_id = $1
   AND status IN ('pending', 'unknown');   -- legal predecessors only
-- 0 rows updated => someone else already moved it; re-read and stop.

ALLOWED = {
    "created":    {"pending", "failed"},
    "pending":    {"authorized", "captured", "failed", "unknown"},
    "unknown":    {"authorized", "captured", "failed"},
    "authorized": {"captured", "failed"},      # failed = auth voided/expired
    "captured":   {"partially_refunded", "refunded"},
    "partially_refunded": {"partially_refunded", "refunded"},
}

def transition(conn, payment_id, new_status, source, raw=None):
    with conn.transaction():
        row = conn.fetchone(
            "SELECT status FROM payments WHERE payment_id=%s FOR UPDATE",
            (payment_id,))
        cur = row["status"]
        if cur == new_status:
            return cur                       # duplicate callback, no-op
        if new_status not in ALLOWED.get(cur, set()):
            raise IllegalTransition(cur, new_status)
        conn.execute("UPDATE payments SET status=%s, updated_at=now() "
                     "WHERE payment_id=%s", (new_status, payment_id))
        conn.execute("INSERT INTO payment_events(payment_id, from_status, "
                     "to_status, source, raw_response) VALUES (%s,%s,%s,%s,%s)",
                     (payment_id, cur, new_status, source, raw))
        conn.execute("INSERT INTO outbox(topic, key, payload) VALUES "
                     "('payment-events', %s, %s)",
                     (payment_id, {"to": new_status}))
    return new_status

Note the terminal states. FAILED is terminal for the payment, but if reconciliation later discovers the customer was actually debited, we do not resurrect the payment; we record a late success event and either capture it (if the order is still open) or auto-refund it. Mutating terminal history makes audits impossible.

Deep Dive 2: Idempotency Keys That Actually Work

The classic failure: the merchant's server calls POST /payments, our server processes it, and the response is lost because of a network blip. The merchant retries. Without idempotency, the customer gets two collect requests or is charged twice.

The algorithm

1. Insert the key first. INSERT INTO idempotency_keys (merchant_id, idem_key, request_hash, locked_at) VALUES (...). The primary key makes this atomic.
2. On conflict, inspect the existing row. If request_hash differs, return 422 (same key, different body is a client bug). If a response is stored, return it verbatim. If it is locked and not finished, return 409 "request in progress" so the client retries later.
3. Do the work (create payment, call the provider).
4. Store the response on the key row and clear the lock, ideally in the same DB transaction as the payment row creation.
5. Expire keys after a window (24 hours to a few days), long enough to cover any reasonable retry.

import hashlib, json

def create_payment(req, merchant_id):
    key = req.headers.get("Idempotency-Key")
    if not key:
        return 400, {"error": "Idempotency-Key required"}
    body_hash = hashlib.sha256(
        json.dumps(req.json, sort_keys=True).encode()).hexdigest()

    with db.transaction() as tx:
        inserted = tx.execute(
            """INSERT INTO idempotency_keys (merchant_id, idem_key,
                   request_hash, locked_at)
               VALUES (%s, %s, %s, now())
               ON CONFLICT DO NOTHING""",
            (merchant_id, key, body_hash)).rowcount == 1
        if not inserted:
            row = tx.fetchone(
                "SELECT request_hash, response_code, response_body "
                "FROM idempotency_keys WHERE merchant_id=%s AND idem_key=%s",
                (merchant_id, key))
            if row["request_hash"] != body_hash:
                return 422, {"error": "key reused with different request"}
            if row["response_code"] is None:
                return 409, {"error": "request in progress, retry later"}
            return row["response_code"], row["response_body"]

        payment = insert_payment(tx, merchant_id, req.json)   # status=created
        response = (201, {"payment_id": payment.id, "status": "pending"})
        tx.execute(
            "UPDATE idempotency_keys SET response_code=%s, response_body=%s "
            "WHERE merchant_id=%s AND idem_key=%s",
            (response[0], json.dumps(response[1]), merchant_id, key))

    submit_to_provider_async(payment.id)   # via outbox, outside this txn
    return response

Notice the provider call happens after commit, via the outbox. If you call the bank inside the DB transaction and the transaction then rolls back, you have a real-world side effect with no record of it. The provider call itself must also be idempotent: we send our payment_id as the transaction reference to the PSP/acquirer, and a retried submission with the same reference is rejected as a duplicate by the provider rather than executed twice.

Deep Dive 3: The Double-Entry Ledger

A naive design stores merchant.balance and does balance = balance + 500. That fails audits, races under concurrency and offers no explanation when the number is wrong. Payment companies use double-entry bookkeeping: every transaction is a set of entries whose debits equal credits.

Worked example: a ₹500 UPI payment with 2% MDR

(UPI person-to-merchant payments are typically zero-MDR in India for most cases, but gateways charge merchants a platform fee; we use a fee here to show the mechanics.)

(GST on the fee would add one more credit line to a tax-payable account; omitted for brevity.) After all three events, upi_receivable is zero (the bank paid what it owed), merchant_payable is zero (we paid the merchant), and ₹10 sits in fee revenue with ₹10 left in the nodal account to match it. If upi_receivable:psp_bank is non-zero at the end of a settlement cycle, reconciliation has found a missing settlement. That is the power of the model: anomalies show up as non-zero balances in accounts that should be zero.

Ledger rules

• Append-only. Never update or delete an entry. Mistakes are fixed with reversing entries.
• Balanced per transaction. The ledger service rejects a write where sum(D) ≠ sum(C), checked inside the DB transaction.
• Balances are derived. Balance = sum of credits minus debits (sign depends on account type). For speed, maintain a cached balance row updated in the same transaction as entries, and periodically verify it against the sum.
• Idempotent ingestion. The ledger consumes payment events from Kafka; a unique constraint on (source_event_id) makes replays harmless.

public void post(String txnId, String sourceEventId, List<Entry> entries) {
    long debits = entries.stream().filter(e -> e.direction() == 'D')
                         .mapToLong(Entry::amountPaise).sum();
    long credits = entries.stream().filter(e -> e.direction() == 'C')
                          .mapToLong(Entry::amountPaise).sum();
    if (debits != credits || debits == 0) {
        throw new UnbalancedTransactionException(txnId, debits, credits);
    }
    jdbi.useTransaction(h -> {
        int ins = h.createUpdate(
            "INSERT INTO ledger_txns(txn_id, source_event_id) VALUES (:t, :s) " +
            "ON CONFLICT (source_event_id) DO NOTHING")
            .bind("t", txnId).bind("s", sourceEventId).execute();
        if (ins == 0) return;                       // replayed event: already posted
        for (Entry e : entries) {
            h.createUpdate("INSERT INTO ledger_entries(txn_id, account_id, " +
                           "direction, amount_paise) VALUES (:t,:a,:d,:m)")
             .bind("t", txnId).bind("a", e.accountId())
             .bind("d", String.valueOf(e.direction()))
             .bind("m", e.amountPaise()).execute();
        }
    });
}

Hot accounts

Accounts like fee_revenue or a single giant merchant receive thousands of entries per second, and a single balance row becomes a lock hotspot. Fixes: don't maintain a live balance for system accounts (compute on demand or in batches), or split the account into N sub-accounts (fee_revenue#0..#15) and sum them when reading.

Deep Dive 4: Timeouts, Retries and Webhooks

The pending problem

Suppose we submit a UPI request and the PSP bank's response does not arrive within 30 seconds. Three things could have happened: the request never reached NPCI, the debit failed, or the debit succeeded and the response was lost. We cannot tell. The correct move is:

1. Mark the payment UNKNOWN, not failed. Tell the customer "payment is processing" instead of "failed", because a "failed" message makes them pay again.
2. Poll with backoff. The status poller calls the provider's transaction status API (for UPI, a check-status request on the original transaction reference) at increasing intervals: 5 s, 15 s, 45 s, then minutes.
3. Accept late callbacks. The bank may call back asynchronously; the state machine handles whichever arrives first and ignores the duplicate.
4. Fall back to reconciliation. If still unknown after the polling window, the daily settlement file decides. Unresolved debited-but-failed cases are reversed. RBI's turnaround-time framework for failed digital transactions specifies time limits for auto-reversal and compensation if they are missed, so this is not just good UX.

Retrying safely

• Retry only idempotent operations, with the same reference ID. Never generate a fresh transaction ID for a retry of the same attempt.
• Use exponential backoff with jitter and a retry budget so a degraded bank does not get hammered by synchronized retries.
• Use per-provider circuit breakers: if a bank's success rate collapses, the router stops sending new traffic to it and picks an alternative route (another PSP bank or acquirer). Rate limiting at the edge (see rate limiter design) protects you from merchant retry storms.

Webhooks

Webhooks are how merchants learn about final states. The dispatcher consumes payment events and POSTs to the merchant's URL.

• At-least-once delivery. Retry on non-2xx with backoff for up to a day or so, then mark the endpoint as failing and alert the merchant. Merchants must dedupe on event_id.
• Ordering is not guaranteed across retries, so include the full current state and a timestamp/version in each event; the merchant applies it only if newer.
• Signed bodies. HMAC-SHA256 over the raw body with a per-merchant secret; merchants verify in constant time.
• Isolation. One merchant with a slow endpoint must not delay others. Use per-merchant queues or partitions and a bounded concurrency per endpoint.

import hmac, hashlib

def verify_webhook(raw_body: bytes, signature_hex: str, secret: bytes) -> bool:
    expected = hmac.new(secret, raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature_hex)

Also tell merchants never to trust the browser redirect ("payment success" page) alone. The redirect can be forged or lost; the webhook or a server-side status fetch is the source of truth.

Deep Dive 5: Reconciliation

Reconciliation compares three views of the same money: our payments table, our ledger, and external records (NPCI/bank settlement reports for UPI, acquirer settlement files for cards, bank statements for the nodal/escrow account). It runs per settlement cycle and daily.

How the job works

1. Ingest external files (SFTP or API), normalise into a common schema keyed by provider reference (RRN/UPI transaction ID) and our payment_id.
2. Join against internal records for the same window, with a tolerance window at the edges (a payment at 23:59:58 may appear in the next day's file).
3. Classify each record: matched, missing-internal, missing-external, mismatched.
4. Auto-resolve the known patterns, push the rest to an ops queue, and block settlement for unresolved items.
5. Post ledger adjustments for every resolution, so the ledger stays the single explanation of balances.

Settlement to merchants should run only on reconciled funds. Paying a merchant for a payment that the bank later reports as never received is the most expensive bug in the system.

Bottlenecks and Trade-offs

Bottlenecks to call out: hot merchants (shard further), hot ledger accounts (sub-accounts), provider rate limits (queue and smooth), and the poller itself during a bank outage, when thousands of payments become unknown at once; the poller needs its own rate limiting per provider so it doesn't worsen the outage.

What Interviewers Look For

Typical follow-ups and short answers:

• "How do you handle a refund?" Refund is its own entity with its own state machine and idempotency key. Ledger: debit merchant_payable, credit refund_payable; when the bank confirms, debit refund_payable, credit the nodal account. Reject refunds exceeding captured minus already-refunded, checked under a row lock on the payment.
• "Two payment attempts for one order both succeed?" The partial unique index makes the second capture fail; that attempt goes to auto-refund.
• "How do you migrate a shard without downtime?" Dual-write via CDC to the new shard, backfill, verify counts and ledger sums, flip reads, then writes per merchant.
• "What about exactly-once?" End-to-end exactly-once across banks is not achievable; we get effectively-once via idempotent operations, dedupe, and reconciliation as a safety net.

Common Mistakes in Payment System Design Interviews

• Floats for money. 0.1 + 0.2 ≠ 0.3. Use integer paise or a decimal type.
• Marking timeouts as failures. The single most damaging bug; causes double payments by anxious customers and lost revenue for merchants.
• Calling the bank inside a DB transaction or publishing events outside one (dual write). Use the outbox.
• Mutable balances with no history. No auditor, and no on-call engineer, can explain the number later.
• Trusting the client redirect or an unsigned webhook as proof of payment.
• Forgetting reconciliation or treating it as a reporting feature. It is the mechanism that guarantees correctness when everything else is only eventually consistent.
• Designing for 1 million TPS and spending 20 minutes on caching. The interesting problems here are correctness, not raw scale.

If you can draw the UPI sequence, the state machine with an unknown state, the idempotency table and a balanced ledger for a payment and a refund, you have covered what most interviewers mean by "design a payment system". Practise saying the failure scenarios out loud; under pressure, that is the part candidates skip. For adjacent practice, the flash sale checkout problem reuses the same reservation and idempotency ideas at the inventory layer.

Frequently Asked Questions

Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.