Design a Distributed Logging System (ELK at Scale)

"Design a logging system" sounds like plumbing, but it is one of the better system design questions because it tests almost everything at once. It is a write-heavy firehose, it needs a durable buffer, it needs search, and its storage bill grows every day. Most of the hard decisions are about cost versus searchability, and that is where strong candidates separate themselves.

This walkthrough designs a centralized logging platform for a company running about 20,000 containers across several hundred microservices, roughly what a large Indian product company like Flipkart, Swiggy or PhonePe runs. We follow the standard order: requirements, estimates, API, data model, architecture, deep dives, trade-offs, and what interviewers actually probe.

Step 1: Clarify Requirements for the Logging System

Start by asking who the users are. Here they are internal engineers debugging production, on-call responders during incidents, the security team, and auditors. Each group wants something different, and that drives the design.

Functional requirements

• Collect logs from every host, container and serverless function, including stdout/stderr and log files.
• Search by time range, service, environment, host, log level, trace ID and free text ("NullPointerException in checkout").
• Live tail: stream matching logs in near real time, like tail -f across a whole fleet.
• Aggregations: error counts per service per minute, top exception messages.
• Retention policies per tenant or log class: debug for 3 days, application logs for 30 days, audit logs for years.
• Alerting hooks on log patterns (for example, more than 100 "payment timeout" lines per minute).

Non-functional requirements

• Ingestion latency: a line should be searchable within about 10–30 seconds at p99. Live tail should be under about 5 seconds.
• Durability: application logs at least once; small loss is tolerable under extreme overload if it is visible. Audit logs must not be lost.
• Availability: ingestion must never block the application. If the logging system is down, the checkout service must keep selling.
• Query latency: a few seconds for searches over the last 24 hours; slower is acceptable for month-old data.
• Multi-tenancy: one noisy service must not starve others.
• Compliance: no card numbers, OTPs or Aadhaar numbers in searchable logs. India's DPDP Act and PCI-DSS both push you towards redaction at ingestion.

Step 2: Capacity Estimation (Show the Math)

Interviewers want to see you turn vague scale into concrete storage and throughput, because those numbers decide whether full-text indexing is affordable. If you want to practise this skill on its own, see back-of-the-envelope estimation for system design.

The "index ratio" depends heavily on mappings. A fully indexed JSON document with _source stored and several keyword fields often ends up roughly the same size as the raw input, sometimes smaller with best_compression, sometimes larger with many analysed text fields. State it as an assumption, not a fact.

The punchline: 600 TB of replicated SSD just for one week of hot search. That number is what makes sampling, field-level indexing decisions and tiering central to this design, not optional extras. Saying this out loud is a strong signal in an interview.

Indexer sizing follows from the same numbers. If one Elasticsearch data node sustains, say, 20–40 MB/s of indexing for this kind of document (benchmark it, it varies a lot with mappings), then 1.5 GB/s at peak needs roughly 40–75 hot nodes just for write throughput, before replicas. That is why many teams index peak traffic with some lag and let Kafka absorb the difference.

Step 3: API Design

Most logs arrive through agents, not a public API, but you still need an ingestion endpoint for serverless functions, mobile backends and third-party systems, plus a query API that the UI uses.

POST /v1/logs:batch
Headers: Authorization: Bearer <service-token>, Content-Encoding: zstd
Body: { "resource": {"service": "checkout", "env": "prod", "region": "ap-south-1"},
        "events": [ {"ts": "2026-10-10T09:15:02.123Z", "level": "ERROR",
                     "msg": "payment timeout", "trace_id": "4bf9...", "attrs": {...}} ] }
Response: 202 Accepted   (or 429 with Retry-After when the tenant is over quota)

POST /v1/search
Body: { "query": "level:ERROR AND service:checkout AND \"payment timeout\"",
        "from": "2026-10-10T09:00:00Z", "to": "2026-10-10T10:00:00Z",
        "limit": 200, "sort": "ts:desc", "cursor": null }
Response: { "hits": [...], "next_cursor": "...", "took_ms": 840, "partial": false }

POST /v1/aggregate
Body: { "query": "level:ERROR", "group_by": ["service"], "interval": "1m", "from": ..., "to": ... }

GET  /v1/tail?query=service:checkout%20AND%20level:ERROR     (WebSocket or SSE)

Two details matter. Ingestion returns 202 Accepted, not 200, because the line is durable in the buffer but not yet searchable. And search returns a partial flag, because a query fanned out to 500 shards may get answers from only 497 within the timeout, and the UI should say so rather than silently hide data.

Step 4: The Log Event Data Model

Push teams towards structured logging: JSON with a fixed envelope and a free-form attributes map. Free-text lines like "user 42 paid 499" force expensive regex parsing downstream; {"user_id":42,"amount":499} does not.

{
  "ts": "2026-10-10T09:15:02.123456Z",     // event time, set by the app
  "ingest_ts": "2026-10-10T09:15:03.010Z", // set by the agent, used to detect lag/skew
  "service": "checkout-api",
  "env": "prod",
  "region": "ap-south-1",
  "host": "ip-10-2-33-14",
  "pod": "checkout-api-7d9f8-xk2lp",
  "level": "ERROR",
  "msg": "payment timeout after 3000ms",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "source_seq": 918273,                    // per-source counter for ordering and dedup
  "sample_rate": 1,                        // 10 means "1 of every 10 kept"
  "attrs": { "order_id": "OD123", "psp": "razorpay", "latency_ms": 3004 }
}

(Comments are for illustration; real JSON has none.) Note the event ID is derivable as hash(host, pod, source_seq). Using that as the Elasticsearch document _id makes re-delivered duplicates overwrite themselves instead of appearing twice, at some cost in indexing speed because Elasticsearch must check for an existing ID. Many teams accept duplicates for debug logs and only enforce IDs for audit logs.

The attrs map is dangerous. If every team can invent keys freely and you use dynamic mapping, you get mapping explosion: tens of thousands of fields in one index, which bloats cluster state and slows the master. Mitigate with an allow-listed set of indexed fields, a flattened field type for the rest, and a per-index field limit.

Step 5: High-Level Architecture

The flow in one sentence: agents tail files and push compressed batches to Kafka, stateless processors clean the stream into a second topic, indexers bulk-load it into time-partitioned Elasticsearch indices, lifecycle management ages indices down through cheaper tiers, and a query service fans searches out across the tiers. An archiver writes everything to S3 as Parquet in parallel, so the search cluster is never the system of record.

Deep Dive 1: Log Collection Agents

In Kubernetes, apps write to stdout and the container runtime writes those streams to files under /var/log/containers/ on the node. A DaemonSet agent (Fluent Bit, Vector, Filebeat or the OpenTelemetry Collector) runs one pod per node, tails those files and ships them. Sidecar agents per pod give more isolation but cost far more memory across 20,000 pods.

What a good agent does

1. Tail and checkpoint. It tracks (file inode, byte offset) in a local checkpoint file. After a restart it resumes from the offset, which gives at-least-once delivery. Tracking the inode, not just the path, handles log rotation correctly.
2. Handle multi-line events. A Java stack trace is 40 lines but one event. The agent joins lines using a start pattern (a line beginning with a timestamp) and a flush timeout.
3. Attach metadata. Pod name, namespace, node, labels and deployment version from the Kubernetes API, cached locally so you do not hammer the API server.
4. Batch and compress. Send batches of 1 MB or every 1 second, whichever comes first, compressed with zstd or lz4. Batching is the biggest single throughput lever.
5. Buffer locally. If Kafka is unreachable, spill to a bounded on-disk buffer (for example 2 GB per node). When the buffer fills, drop the oldest debug lines first and emit a "dropped N lines" counter as a metric.
6. Respect limits. The agent gets a CPU and memory cap. A logging agent that eats 2 cores during an incident is itself an incident.

Deep Dive 2: Why Kafka Sits in the Middle

Without a buffer, agents write straight into Elasticsearch. That works at small scale and breaks in exactly the wrong moment: during an outage, every service starts logging errors, volume triples, Elasticsearch indexing slows down under merge pressure, agents back up, and you lose the logs you most need. Kafka fixes this by being a durable, replayable queue that is cheap to write to sequentially. If you are new to how Kafka achieves this, read how to design a distributed message queue like Kafka.

Topic and partition design

• Topics by class: logs.raw.app, logs.raw.audit, logs.raw.security. Audit uses acks=all and min.insync.replicas=2; debug-heavy app logs can use acks=1 to trade a little durability for latency.
• Partition key: you rarely need strict ordering for logs, so spread load with a sticky or round-robin partitioner. Keying by service would create hot partitions, because one chatty service can produce 30% of all lines.
• Partition count: at ~100 MB/s of compressed peak traffic and a comfortable 5–10 MB/s per partition for consumers, you need tens of partitions; teams usually provision a few hundred so they can scale indexer consumers without repartitioning.
• Retention: 72 hours. That is the window in which you can fix a broken indexer and replay without losing a single line.

Kafka also enables fan-out: the indexer, the S3 archiver, the live-tail service, the security SIEM and a pattern-alerting job all consume the same topic with separate consumer groups, each at its own pace. The trade-offs versus other brokers are covered in Kafka vs RabbitMQ vs SQS; for logs, Kafka's replayable log is the deciding feature.

Deep Dive 3: Processing, Enrichment and PII Redaction

A stateless processor tier (Logstash, Vector in aggregator mode, or a Flink/Kafka Streams job) reads logs.raw.* and writes logs.clean.*. Keeping processing separate from indexing means a parsing bug does not require redeploying the indexers, and you can replay raw data through fixed parsers.

• Parse: JSON lines pass through; legacy plain-text lines go through grok/regex patterns per service. Unparseable lines are kept with parse_error=true rather than dropped.
• Normalize: map severity, lvl, loglevel to one level field; convert timestamps to UTC.
• Enrich: add team ownership, cost centre and on-call rotation from a service catalogue cached in memory.
• Redact: mask 16-digit card numbers (with a Luhn check to avoid masking order IDs), 12-digit Aadhaar patterns, OTPs, phone numbers, emails and anything in fields named password, token or authorization. Redaction must happen before indexing and before archiving, because deleting a leaked value from an immutable Lucene segment or a year of Parquet files is painful.
• Route and sample: decide the destination index and apply sampling rules (covered below).

Deep Dive 4: Indexing into Elasticsearch / OpenSearch

Elasticsearch stores each index as shards, and each shard is a Lucene index made of immutable segments with an inverted index per field. If you need a refresher on why inverted indices make "find every line containing timeout" fast, see Elasticsearch and the inverted index explained.

Time-based indices with rollover

Logs are append-only and queried by time, so you never update documents and you delete whole time ranges. That makes time-partitioned indices ideal: logs-app-2026.10.10-000001, rolled over when the primary shard reaches a target size or age. Deleting old data becomes "drop the index", which is instant, instead of "delete by query", which is expensive.

Shard sizing

Elastic's own guidance is to aim for shards of roughly 10–50 GB and to avoid huge numbers of tiny shards, because every shard costs heap and cluster-state overhead. At 43 TB/day, a daily index would need on the order of 1,000 primary shards; rollover by size (say 50 GB per primary shard) keeps shards uniform no matter how traffic swings across the day.

Mappings that control cost

Indexers use the bulk API with batches of 5–15 MB, commit Kafka offsets only after a bulk request succeeds (at-least-once), and back off when Elasticsearch returns 429 rejections. Setting refresh_interval to 15–30 seconds instead of the default 1 second noticeably improves indexing throughput, and it matches the 10–30 second searchability target from the requirements. That is a nice moment to connect a design knob to a requirement.

Deep Dive 5: Hot, Warm, Cold and Frozen Storage Tiers

About 80–90% of log queries in most organisations target the last day or two, because people are debugging something that is happening now. That access pattern is what tiering exploits. Index Lifecycle Management (ILM) policies move indices automatically.

Two ideas make this cheap. First, force-merge on warm indices reduces segment count, which shrinks storage and speeds searches on data that will never change again. Second, searchable snapshots let the cold and frozen tiers skip replicas entirely, because the copy in object storage is already durable (S3 is designed for eleven nines of durability). For how object stores achieve that, see designing object storage like S3.

Redo the cost math with tiers. Keeping only 3 days on hot SSD with one replica is about 43 × 3 × 2 ≈ 260 TB instead of 600 TB, warm holds 11 days on cheaper disks, and everything older is mostly in S3. The exact rupee saving depends on your cloud contract, so in an interview just show that the expensive tier shrank by more than half.

Deep Dive 6: Query Path and Live Tail

Search

1. Authorise. The query service checks which services and environments the user may see and appends those as filters. Security logs are visible only to the security team.
2. Prune by time. It maps the time range to the minimal set of indices using index naming or a metadata catalogue. A 1-hour query should touch a handful of indices, not 90 days of them.
3. Fan out. Elasticsearch's coordinating node sends the query to every relevant shard; each returns its top N by timestamp; the coordinator merges. Cross-cluster search lets one query span the hot cluster and the frozen tier.
4. Guard rails. Cap time range for unfiltered free-text queries, reject leading wildcards like *timeout on large ranges, set per-query timeouts and return partial results.
5. Paginate with cursors (search_after on ts plus a tie-breaker), not deep from/size offsets, which get slower the deeper you page.

Live tail

Do not implement live tail by polling Elasticsearch every second; that waits for refresh and hammers the cluster. Instead, a tail service consumes logs.clean.* from Kafka, evaluates each active tail filter in memory and pushes matches to browsers over WebSocket or Server-Sent Events. The comparison of those transports is in WebSockets vs SSE vs long polling; SSE is usually enough because traffic is one-way. Rate-limit each tail session (for example 500 lines/s) so nobody streams the entire firehose into a laptop.

Log-based alerting

A streaming job counts matches of registered patterns per minute per service and pushes those counts to the metrics system, where normal alert rules apply. Turning logs into metrics early is much cheaper than running a search every minute. The alerting side is designed in detail in metrics monitoring and alerting system design.

Deep Dive 7: Sampling and Cost Control

This section is where senior candidates shine, because it is where real logging platforms spend most of their engineering time.

Sampling rules that do not destroy debuggability

• Never sample ERROR, WARN, audit or security events.
• Per-service, per-level rates: keep 100% of INFO for low-volume services but 10% for a service emitting 200k lines/s of access logs. Store sample_rate on each kept event so dashboards can multiply counts back up.
• Trace-consistent sampling: decide keep/drop by hash(trace_id) mod 100 < rate. Every service then keeps the same requests, so a sampled request is complete end to end instead of having random holes.
• Deduplication: collapse identical messages from the same pod within a short window into one event with repeat_count=4,312. Retry storms produce exactly this pattern.
• Dynamic overrides: on-call can raise a service to 100% for 30 minutes during an incident. Rules live in a config store that agents and processors poll.

Quotas and multi-tenancy

Each team gets a daily ingest quota tied to its cost centre. Over quota, the pipeline first samples that team's INFO/DEBUG harder, then drops them, but always keeps errors. Showing teams a weekly bill for their logs changes behaviour faster than any guideline. A token bucket per tenant at the ingest gateway is the usual mechanism; the algorithm is explained in rate limiter system design.

Change what you index

A practical hybrid: ERROR/WARN and the last 3 days of INFO go to Elasticsearch for fast search; everything goes to a cheap label-indexed or columnar store for longer, slower queries. Being able to articulate this hybrid, with the cost reason, is a clear senior signal.

Bottlenecks, Failure Modes and Trade-offs

The key trade-off to state: durability versus cost and latency. You could make every log line as durable as a payment record, with synchronous replication and no sampling, but the bill would be enormous for data that is mostly never read. A tiered guarantee (audit logs: never lose; app logs: at-least-once with bounded loss under extreme overload; debug: best-effort and sampled) is the mature answer.

What Interviewers Look For in a Logging System Design

Follow-up questions to prepare

• "How would you find all logs for one user request across 15 services?" Propagate a trace ID (W3C traceparent) in headers and log it on every line; search by trace_id.
• "A team wants to keep logs for 7 years for audit." Separate audit topic, no sampling, acks=all, archive to object storage with object lock (WORM), query by re-hydration.
• "How do you delete a user's data on request?" Hard for immutable segments. Avoid logging personal data, store only pseudonymous IDs, and keep a short retention for anything that might contain it.
• "Elasticsearch is too expensive; what now?" Walk through the Loki/ClickHouse hybrid above.
• "How do you know you are losing logs?" Agents and processors emit sent/dropped counters per source; compare with indexed counts per minute and alert on the gap.

Common Mistakes Candidates Make

• Agents writing directly to Elasticsearch with no buffer. Fine for a startup, wrong at the scale you were given.
• One giant index instead of time-based indices, which makes retention a delete-by-query nightmare.
• Keying Kafka partitions by service and creating hot partitions, when logs do not need strict ordering.
• Ignoring cost entirely. If you never compute storage, you will never get to the most interesting part of the problem.
• Sampling errors. Errors are the logs people actually read; sample the noise, not the signal.
• Polling Elasticsearch for live tail instead of reading the stream.
• Forgetting PII. In a payments or fintech interview (Razorpay, PhonePe, Paytm), skipping redaction is a red flag.

If you can draw the pipeline in two minutes, defend Kafka's place with an outage scenario, compute the hot-tier size, and then cut it in half with tiers and sampling, you have given a better answer than most candidates. For a broader list of problems to practise next, browse system design interview questions for India.

Frequently Asked Questions

Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.