Design Object Storage like Amazon S3

Amazon S3 holds a staggering amount of the internet's data: images, backups, logs, data lakes, ML datasets, static websites. "Design S3" (or "design a blob store", "design Azure Blob / GCS") is a favourite senior-level system design question because it tests storage fundamentals rather than product features: how bytes survive disk failures, how you index trillions of names, and what "consistent" means when a write touches many machines.

This guide uses S3's public behaviour and published documentation as the target, and a generic design for the internals. S3's actual internal implementation is not fully public, so I will be clear about which parts are "how you would build it" rather than "how AWS built it".

The S3 Data Model in Two Minutes

• Bucket: a globally named container with a region, policies and settings (versioning, lifecycle, encryption).
• Object: bytes plus metadata (size, content type, user metadata, checksum/ETag), identified by key within a bucket. Objects can be very large; AWS documents a maximum object size of 5 TB.
• Flat namespace: photos/2026/10/a.jpg is one key. "Folders" are an illusion created by listing with prefix=photos/2026/ and delimiter=/.
• Immutable objects: you cannot append or modify bytes in place; a PUT replaces the whole object (or creates a new version).

Immutability is the gift that makes the whole design tractable: data files never need in-place updates or locking. Only the small metadata record changes.

Requirements

Functional

• Create/delete buckets; PUT, GET (including byte-range GET), HEAD, DELETE objects.
• LIST objects by prefix with a delimiter, sorted lexicographically, paginated.
• Multipart upload for large objects.
• Versioning per bucket, with delete markers.
• Access control (bucket policies, presigned URLs), encryption at rest.
• Lifecycle rules: transition to colder tiers, expire old versions, abort incomplete uploads.

Non-functional

• Durability: extremely high (the S3 Standard design target is 11 nines). Losing data is the worst possible failure.
• Availability: high, but lower priority than durability (S3 Standard is designed for 99.99%).
• Strong read-after-write consistency for object PUT/DELETE and LIST.
• Throughput scaling: aggregate bandwidth grows with the number of clients; individual prefixes must handle thousands of requests per second.
• Low cost per GB: the economics drive the design toward erasure coding and dense disks.

Capacity Estimates

Interview numbers for a large regional deployment:

• 100 billion objects, average size 1 MB (heavily skewed: most objects are small, most bytes are in large objects) → 100 PB logical data.
• With 3x replication: 300 PB raw. With 6+3 erasure coding (1.5x): 150 PB raw. At 20 TB per disk, that is 7,500 disks vs 15,000; at ~60 disks per storage server, 125 servers vs 250. Before free-space headroom.
• Metadata: ~1 KB per object (key, version, size, checksum, ACL, placement) → 100 TB of metadata, replicated 3x for consistency. Too big for one machine; must be partitioned.
• Requests: assume 1M requests/s peak, 80% GET. Every request hits metadata; only GET/PUT hit the data plane.
• Bandwidth: if 200k GET/s average 1 MB, that is 200 GB/s egress, so the data plane is bandwidth-bound and must spread hot objects across many disks.
• Disk failures: with ~1% annual failure rate across 7,500 disks, expect ~75 failures a year, more than one a week. Repair is a continuous background process, not an incident.

API

Presigned URLs matter in real architectures: an Instagram-style app does not stream uploads through its own API servers. The server signs a URL, the phone uploads directly to object storage. You will see this pattern in Google Drive/Dropbox design and YouTube design.

High-Level Architecture

Four independent concerns, each scaled separately:

1. API tier: stateless, terminates HTTPS, authenticates, streams bytes. Scales with request rate and bandwidth.
2. Metadata service: the index. Scales with object count and request rate.
3. Placement service: knows disk health, capacity and topology; chooses where new fragments go so each stripe spans failure domains.
4. Storage nodes: dumb, dense, reliable byte stores. Scale with bytes.

The Metadata Plane

What a metadata record holds

key:   (bucket_id, object_key, version_id_desc)
value: {
  size: 734003200,
  etag: "9b2cf535f27731c974343645a3985328",
  checksum: {algo: "CRC32C", value: "..."},
  content_type: "video/mp4",
  user_meta: {...},
  storage_class: "STANDARD",
  is_delete_marker: false,
  created_at: 1760000000123,
  layout: [                      # where the bytes are
    {stripe: "s-81f2", ec: "6+3", offset: 0,        length: 402653184},
    {stripe: "s-91aa", ec: "6+3", offset: 402653184, length: 331350016}
  ]
}

Note version_id_desc: storing versions in descending order (newest first) means "get the latest version" is a single seek to the first row with that key prefix.

Partitioning the metadata

100 TB of metadata and ~1M requests/s needs hundreds of partitions. The choice of partitioning scheme is the most interesting decision here:

Because LIST with prefix is a core API, object stores generally use range partitioning with automatic splitting of hot or large ranges, similar to Bigtable tablets. This is also why AWS documents per-prefix request rates (at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix) and notes that S3 scales by partitioning prefixes as load grows. Partition-splitting strategies are discussed in sharding strategies.

Consistency of metadata

Each partition is a replicated state machine (Raft or Paxos) across three availability zones. Writes commit when a majority acknowledges; reads go to the leader (or use leases) so they see the latest committed write. That is what delivers strong read-after-write consistency: once a PUT returns 200, the metadata commit is in a majority, and any subsequent GET or LIST is served from the up-to-date state. Caches in front of metadata must be invalidated or versioned to preserve this property, which is the hard part in practice.

The Data Plane and the Write Path

Storage layout

Storage nodes do not create a filesystem file per object. They manage large append-only extents (say 256 MB to 1 GB) and write fragments into them, keeping a small local index of offsets. Small objects are packed together; large objects span many stripes. This is the same idea as Facebook's Haystack paper for photo storage: avoid filesystem metadata overhead for billions of tiny files.

PUT, step by step

1. Authenticate and authorise the request at the API tier.
2. Ask placement for a stripe: 9 storage nodes for 6+3 erasure coding, on different disks, racks and (for regional durability) spread across three availability zones.
3. Stream and encode: the API node reads the body in chunks, computes the checksum incrementally, splits each chunk into 6 data fragments, computes 3 parity fragments, and streams all 9 to storage nodes in parallel.
4. Durable acknowledgment: each storage node fsyncs and acknowledges. Wait for all 9 (or a write quorum like 8 of 9 with background repair of the missing one, depending on your policy).
5. Verify checksum against the client's Content-MD5/checksum header. Mismatch → fail; the written fragments become garbage.
6. Commit metadata: write the record (key, version, layout) to the metadata partition. This is the commit point. Before it, the object does not exist; after it, it is visible to everyone.
7. Return 200 with the ETag.

GET

1. Look up metadata (latest version, or the requested one). Check it is not a delete marker.
2. For the requested byte range, compute which stripes and fragments are needed.
3. Read the 6 data fragments in parallel (no decode needed in the happy path). If one is slow or missing, read a parity fragment and decode instead. Requesting one extra fragment up front and using the first 6 to arrive is a common tail-latency trick.
4. Verify fragment checksums, stream bytes to the client.

Erasure Coding vs Replication

With Reed-Solomon RS(k, m), data is split into k fragments and m parity fragments are computed such that any k of the k+m fragments reconstruct the data. The simplest possible erasure code is single XOR parity (m = 1), as in RAID-5:

def xor_parity(blocks):
    parity = bytearray(len(blocks[0]))
    for b in blocks:
        for i, byte in enumerate(b):
            parity[i] ^= byte
    return bytes(parity)

data = [b"HELLO_", b"WORLD_", b"S3!!!!"]          # k = 3 equal-size data chunks
parity = xor_parity(data)                          # m = 1 parity chunk

lost = 1                                           # the disk holding chunk 1 dies
survivors = [c for i, c in enumerate(data) if i != lost] + [parity]
print(xor_parity(survivors))                       # b'WORLD_' rebuilt

XOR survives one loss. Reed-Solomon generalises this with arithmetic over a finite field (GF(2⁸)) so that m parity fragments survive any m losses. Production systems use optimised libraries such as Intel ISA-L rather than hand-written code.

The real trade-offs

• Repair amplification: rebuilding one lost 1 GB fragment in RS(10,4) reads 10 GB across the network. With a disk failure a week, repair traffic is a permanent background load. Locally Repairable Codes add local parity groups so most repairs read fewer fragments.
• Small objects: erasure-coding a 4 KB object into 9 fragments is wasteful in IOPS and per-fragment metadata. Pack small objects into large extents first, then erasure-code the extents.
• Latency: EC writes fan out to 9 nodes and reads need 6; tail latency depends on the slowest of many. Mitigate with hedged requests.
• Hybrid approach: write new data replicated (fast, simple), then convert to erasure coding in the background once it is sealed and cooled. Many large systems do some variant of this.

The Durability Math

Interviewers love "how many nines does your design give?" Here is a simple model you can do on a whiteboard, followed by the caveat that matters more than the model.

Assumptions: annualised disk failure rate (AFR) of 1% (public drive statistics, such as those Backblaze publishes, commonly show roughly 1 to 2%); a lost fragment is re-protected within 6 hours (detection plus parallel rebuild); failures are independent.

• Probability a given disk fails inside a 6-hour window: p = 0.01 × 6 / 8,760 ≈ 6.8 × 10⁻⁶.
• 3x replication: the first failure among 3 copies happens with rate ≈ 3 × 0.01 = 0.03/year. Data is lost only if the other two both fail within the window: p² ≈ 4.7 × 10⁻¹¹. Annual loss ≈ 0.03 × 4.7 × 10⁻¹¹ ≈ 1.4 × 10⁻¹², about 12 nines.
• RS(6,3): first failure rate ≈ 9 × 0.01 = 0.09. Loss needs 3 more of the remaining 8 within the window: C(8,3) p³ ≈ 56 × 3.2 × 10⁻¹⁶ ≈ 1.8 × 10⁻¹⁴. Annual loss ≈ 1.6 × 10⁻¹⁵, about 15 nines, at half the storage cost.

from math import comb, log10

def annual_loss_probability(n, tolerate, afr=0.01, repair_hours=6.0):
    """Rough model: a stripe of n fragments survives `tolerate` losses.
    After one fragment fails, data is lost if `tolerate` more of the remaining
    n-1 fail within the repair window. Assumes independent failures."""
    p = afr * repair_hours / 8760                 # P(a disk dies inside the window)
    first = n * afr                               # expected first failures per year
    more = sum(comb(n - 1, j) * p**j * (1 - p)**(n - 1 - j)
               for j in range(tolerate, n))
    return first * more

for name, n, t, overhead in [("3x replication", 3, 2, 3.0),
                             ("RS(6,3)", 9, 3, 1.5),
                             ("RS(10,4)", 14, 4, 1.4)]:
    loss = annual_loss_probability(n, t)
    print(f"{name:15s} overhead {overhead:.1f}x  P(loss)/yr {loss:.1e}  ~{-log10(loss):.0f} nines")

# 3x replication  overhead 3.0x  P(loss)/yr 1.4e-12  ~12 nines
# RS(6,3)         overhead 1.5x  P(loss)/yr 1.6e-15  ~15 nines
# RS(10,4)        overhead 1.4x  P(loss)/yr 2.2e-19  ~19 nines

Two lessons fall out. First, repair time is a lever as strong as redundancy: halving the repair window multiplies the 6+3 loss probability by roughly 1/8 (it enters as p³). That is why rebuilds are spread across hundreds of disks in parallel instead of onto a single spare. Second, these numbers are wildly optimistic, because:

Multipart Upload

Uploading a 50 GB database backup in one HTTP request is fragile: a dropped connection at 49 GB means starting over, and a single stream cannot use all available bandwidth. Multipart upload fixes both. AWS documents parts of 5 MiB to 5 GiB (the last part can be smaller), up to 10,000 parts per upload.

1. Initiate: POST /bucket/key?uploads creates an upload record in metadata (upload_id, key, owner, created_at) and returns upload_id. No object is visible.
2. Upload parts in parallel: each PUT ?partNumber=N&uploadId=… is written to the data plane like a small object; metadata stores (upload_id, part_number) → layout, size, ETag. Re-uploading part N replaces it, so failed parts are simply retried.
3. Complete: the client sends the ordered list of (part number, ETag). The service validates every part exists with that ETag and size constraints, then atomically writes one object metadata record whose layout is the concatenation of part layouts. No bytes are copied.
4. Abort or expire: abort deletes the upload record; parts become garbage. Lifecycle rules can abort uploads left incomplete for N days, because forgotten parts quietly cost money.

Note that the ETag of a multipart object is not the MD5 of the whole object; AWS computes it differently for multipart uploads, which regularly confuses people verifying downloads. Use the dedicated checksum features or your own end-to-end hash.

Versioning, Deletes and Consistency

Versioning

With versioning enabled, metadata rows are never overwritten:

• PUT inserts a new row with a new, time-ordered version id.
• GET without version reads the newest row; if it is a delete marker, return 404.
• DELETE without version inserts a delete marker. The data stays.
• DELETE with version id permanently removes that version; its fragments become garbage.
• Lifecycle rules expire non-current versions after N days, which keeps costs bounded.

Versioning plus object lock (write-once-read-many retention) is the standard defence against ransomware and accidental deletes for backups.

Concurrent writes to the same key

Two clients PUT the same key at the same time. Both write data independently; the metadata partition serialises the two commits, and the later commit wins (last-writer-wins). With versioning, both versions are kept. S3 also supports conditional writes with If-None-Match and If-Match headers for compare-and-swap style behaviour, which is useful when you need "create only if absent".

Why strong consistency was hard (and how)

Historically S3 offered eventual consistency for overwrites and LIST, largely because metadata was cached and replicated asynchronously for performance. AWS announced strong read-after-write consistency in December 2020. The general technique is what you would propose: a strongly consistent replicated metadata store as the source of truth, and caches that are invalidated or version-checked in a way that never serves stale data after a successful write. For the trade-offs this implies, see CAP theorem for system design.

LIST implementation

LIST with prefix=photos/2026/&delimiter=/ scans the sorted metadata range starting at the prefix, returning keys and collapsing anything after the next / into "common prefixes" (the folders). A range-partitioned metadata store makes this a sequential scan across one or a few partitions. The continuation token encodes the last key returned, so pagination is stable even as new keys are added.

Integrity, Garbage Collection and Storage Tiers

End-to-end integrity

• Client-provided checksum verified on upload; checksum stored in metadata.
• Per-fragment checksums on storage nodes, verified on every read.
• Background scrubbing: periodically read every fragment and verify checksums to catch silent corruption (bit rot) before a second failure makes it unrecoverable.

Repair

Storage nodes heartbeat to a cluster manager. When a disk or node is declared dead, the repair service enumerates affected stripes (from a stripe-to-disk index), prioritises those with the fewest surviving fragments, and rebuilds fragments onto new disks with rate limits so repair does not starve foreground traffic.

Garbage collection

Orphans come from failed PUTs, overwrites without versioning, deleted versions and aborted multipart uploads. A safe approach is mark and sweep: periodically enumerate all layouts referenced by metadata, compare with extents on storage nodes, and reclaim unreferenced fragments older than a grace period (so an in-flight PUT that has not committed yet is not collected). Compact extents whose live ratio falls below a threshold by copying live fragments into new extents.

Storage classes and lifecycle

Security

• Encryption at rest with envelope encryption: each object (or extent) has a data key, itself encrypted by a master key in a KMS. Rotation re-encrypts data keys, not petabytes of data.
• Presigned URLs: an HMAC signature over method, bucket, key, expiry and headers, computed with the owner's secret; the service recomputes and compares. Short expiries, single purpose.
• Block public access by default; many real-world data leaks are misconfigured public buckets, not breaches.

Interview Checklist and Common Mistakes

What interviewers listen for

Mistakes I see in mock interviews

• Designing a file system: directories as real entities, in-place edits, POSIX semantics. S3 is a flat key-value store of immutable blobs.
• Hash-partitioning metadata and then being unable to explain efficient prefix LIST.
• Storing the bytes in the metadata database (or storing metadata only in storage nodes and scanning them for LIST).
• Making the object visible before data is durable, or never mentioning cleanup of orphaned data.
• Claiming 11 nines from replication alone without discussing repair time, failure domains or checksums.
• Ignoring multipart upload for large objects, or implementing completion by copying parts.

If you want the neighbouring problems, a Dynamo-style key-value store covers the replication and partitioning ideas used in the metadata tier, and back-of-the-envelope estimation will help you produce the capacity numbers above under interview pressure.

Frequently Asked Questions

Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.